Okay, Kolen, please, please take it away.
Hello, oh, wait a second.
Oh, sorry, sorry.
Can you hear me?
Yeah, yeah.
Okay, okay, I will just do it like this.
So can you all see the slides?
Yeah.
Okay, so I will start with a big picture.
So basically, this is an introduction to the SO:UK Data Centre.
And I will start with a big picture.
So first of all, what is HPC and HTC?
So let's define this terminology first.
And this is according to the...
The so-called European Grid Infrastructure (EGI), High Throughput Computing (HTC) is a computer computing
paradigm that focuses on the efficient execution of a large number of loosely coupled tasks.
Given the minimal parallel communication requirements, the task can be executed on cluster, et cetera.
So it's loosely coupled, minimally parallel.
And on the other hand, HPC, high-performance computing, is focusing on the efficient execution
of compute-intensive, tightly-coupled tasks.
Given the high...
Okay.
parallel communication requirements usually involve low latency interconnects that
can share data very rapidly. So I made a table on the right hand side that shows some of the
differences between an HTC kind of computer and HPC kind of computer. So HTC loosely coupled,
HPC tightly coupled, and then interconnect low bandwidth, let's say 10 gigabit ethernet,
versus high bandwidth low latency like InfiniBand interconnects. And then in terms of
computational capability, HTC is a subset of HPC in the sense that any workflow that you can execute
on HTC should be computable on HPC, but not vice versa. In terms of cost,
then HTC has a lower cost per node, hence it has higher throughput per system budget
because each node is cheaper. You can have more nodes, right? So HPC on the other hand is more
expensive. You need to spend more money on interconnects, et cetera. So for example,
in SO:UK Data Centre at a point like we try to estimate how much it will cost for us to build
a high performance, only the storage system for the capacity we need, right? So we need to spend
basically all of our budget just to create the storage system. So it's much more expensive that way.
So in terms of parallelism, HTC is technically possible. You can run MPI jobs there,
but it won't scale very well beyond one node, right? So it means like 10 nodes, 100 nodes,
probably fine. But for HPC, you can scale up to 10,000 nodes without much problem. The biggest
supercomputer on earth is HPC, basically. So homogeneity, HTC is very forgiving in a hetero
homogeneous nodes, which is what we are going to have in SO:UK Data Centre, but HPC is highly
homogeneous. MPI support in HTC often is afterthoughts. In HPC, often it's first class or
maybe the only way you can launch a multi-nodes job, something like that. HTC sometimes is also
known as the grid computing, technically, which is like a subset of HTC are the grid,
and it's usually used by people like high energy physicists, like from people from CERN, et cetera.
Any questions so far? So examples of HPC and HTC in CMB data analysis. So to oversimplify,
to think about like what kind of compute system we need to deploy our scientific application,
we can think in terms of how much memory your application needs, right? So if the amount of
memory is very, very large, beyond one single node, then probably you need HPC. So on the one hand of
the spectrum, which is not on the diagram on the right-hand side, there's something relatively new,
something called so-called Cosmoglobe that some of you already know. So basically it's a full Bayesian
analysis on CMB data, which requires you to load all the TOD in memory, including all the different
frequencies of maps, right? So I estimated basically, if you want to do this kind of analysis
using SO data, you require basically the whole NERSC computer that might barely fit in all the
data or it might actually doesn't. So you might actually need something even bigger.
So, but for typical other kind of like CMB analysis, like maximum likelihood or
medium-based mapmaking, then you need to load all the, at least like per frequency map into the data,
into memory. Then for example, this kind of analysis is done in Planck and will be done in
SO LAT mapmaking. So these are more like HPC kind of workload. So for naive or sometimes called filter
bin or biased mapmaking, this suits the HTC kind of workflow very well because of something called
the MapReduce paradigm. So each mapmaking is a single, can be, can be a single, very small,
I mean, it takes a small chunk of TOD and make a single map out there without
involving any other data. So it's well suited for HTC kind of workflow.
I think I might have missed something here.
Yes. So the thing I want to mention in this slide is actually it has been demonstrated in CMB that
the HTC kind of workflow already is well suited in SPT-3G. So they have done, they actually released
a paper that I tried to cite here. I put it in the last slide actually. And then they demonstrated
they can use the Open Science Grid (OSG) in the US to use HTC kind of resource to complete the analysis
of SPT-3G, right? So there's something already demonstrated to work well. And the next slide is,
I just call that the NERSC problem.
So, you know, you probably, most of you has already received this email in early this year, talking about like the allocation of NERSC resources to the CMB community, right?
So the way I summarize the situation is that typically, if you want to use NERSC resource, you need to make a certain kind of request, right, to request some kind of allocation.
But this is done by someone else for us already for like 25 years.
They routinely, every year, they dedicate 1% of the resources to the whole CMB community.
And then it's also very generously, any CMB-like researcher want to have, I mean, use that system,
We are allocated 1% of that whole CMB pool of resources.
It has been working well.
This model has been working well until very lately.
Basically, it's because of the so-called the end of the Moore's law, right?
So on the chart on the right-hand side, it's kind of complicated.
But basically, the blue line there is the solar blue line is how you would think the supercomputer would grow in time.
But the dotted lines tells you that Moore's law is actually slowing down.
But you can see the other lines basically telling you that how the CMB data sets are growing.
And it hasn't stopped growing by that
trend, right? So it means that the amount of data we are having is increasingly overgrowing the
amount of compute capability we have in supercomputer basically. So this is the sort of problem that we
are facing. So for example, in this year, we know that about 0.1% of the whole NERSC is allocated
to the SO analysis, right? So if you take the amount of FLOPS you have for the whole NERSC
machine multiplied by that number, that's equivalent to a 0.1 PFLOPS machine, basically.
So if you have a 0.1 PFLOPS machine running like for full time for a year, that is the amount
of compute time you have. And by my estimation, SO:UK data centre will have basically the same
similar order of magnitude amount of PFLOPS. So it's an equally capable
machine.
for that allocation. And SO:UK data centre is allocated within Blackett, and which is like
10 times higher, meaning that the amount of burst throughput you can get when you have like you have
a lot of jobs you want to finish very quickly is 10 times as much. So at NERSC is like theoretically
1000 times as much. Here is about 10 times as much. So this slide I call it HTC for SATs. So I
actually mentioned it here. So naïve mapmaking
is well-suited for HTC kind of workflow
because of the MapReduce paradigm.
And it has been demonstrated to use the Open Science Grid,
which is very similar to Blackett here.
And SO:UK Data Centre is funded to perform SO SATs analysis.
And located from within Blackett is an HTC kind of workflow,
and it is funded for eight years.
So in this perspective, it is a very stable long-term commitment
to the science readiness of SAT analysis.
So now I'm going to switch to the next session.
So it's basically I'm soft-launching the SO:UK Data Centre.
What I mean is it's the first time we are presenting
how you can use our resources here.
So what is the SO:UK Data Centre first?
So physically and infrastructurally, it's located from within Blackett.
And which amounts to about 10% of the Blackett resources
in terms of number of CPUs.
And we are going to have access to most of the available resources there.
And in terms of interacting with the computational resources,
we are unique in the sense that typical Blackett users
are not using the resources like that.
They submit their jobs via something called DiracUI,
which we are not going to deal with.
And we are going to log into a certain log-in nodes
or call submit nodes and use HTCondor directly.
So the SO:UK Data Centre documentation is written specifically for this
because no other Blackett users has been using the resources like this,
and HTCondor itself. So I'm just trying to introduce it a little bit.
In a certain sense, you can say it's an inferior job manager
comparing to the rest of the Blackett users.
So the one that you are already used to launch,
in handling massively parallel applications, right?
But we are not doing that in SATs and others.
So it can be viewed as SLURM-like in many aspects.
So basically, it's job manager, you submit jobs,
and it will try to allocate resources for it to run.
However, other design choices in Blackett also contributes
to other differences that you might have been experiencing
with NERSC, for example.
So we are going to see some of them today.
So now, we are going to have a live demo,
which is a little bit scary because live demo,
sometimes it can go wrong.
So let me try.
So the first thing I will show you is, actually, how do I see?
So we are not seeing this.
So let me try to click, just click this thing.
So this is a link to the SO:UK Data Centre documentation.
And for example, some of them is very straightforward.
This is the onboarding page.
So if you have any other colleagues,
you want them to create new users,
you can send them to this page first
and then try to follow the example
to start to be a user of our data centre.
And then there's a quick start page
here that basically highlights the minimal number of pages
you can visit in order to quickly try
to run a certain kind of job.
OK?
And here, this page, I'm trying--
trying to outline what is the typical lifecycle of a workflow or pipeline here, right? So basically,
it's really that short. So you need some way of defining your job configuration. At NERSC,
you have something called, I forgot, batch script. They call it batch script, which has some
special SLURM command on the top of your script, right? So here, we have something called ClassAd.
You write that job configuration, you submit that to the scheduler, and it will try to allocate
the resource. And within that job, then you need to bootstrap a certain software environment.
You need to load the environment that has TOAST, for example, in order to run a mapmaker using TOAST.
And then, optionally, so depending if your application are using MPI or not,
then you can try to launch MPI applications. But as I have briefly mentioned before,
MPI is not a first-class thing in HTCondor, so I provided a special wrapper basically trying to launch a parallel job in something called "parallel universe" in their terminology.
And with that wrapper, you can launch an MPI job, which we are going to see very soon.
And then lastly, you want to do some I/O, right?
So once you have your software, you launch an MPI application, maybe you read in some data, and then it is going to write some data.
Specifically when you're writing data here at Blackett, because it's part of the grid, there's something called the grid storage system.
So we are also going to see how it's going to work over there.
So now it's time for the live demo. So hopefully it works.
So I'm going to switch my screen.
Okay, so are you seeing my terminal?
Yeah.
Okay. So let's try to log into the login nodes first.
So right now we are at this location, so-called VM77. This is now within Blackett.
One thing I didn't mention is that the documentation is actually located at a GitHub repository.
And the code you see in the documentation is actually in a file within the documentation that you can actually run those code.
So the thing I'm going to show you is, let's see, so let's say user document.
So in the website, you can just click them.
So now I'm within the directory, so I'm
going to navigate on my own.
So now I'm in MPI application.
So here are the files.
So first of all, you need to have something
like a job configuration file.
So this is your job configuration file.
So it looks like you try to define what kind of universe
you're in, the so-called parallel universe,
because you work more than one node.
This is the way that HTCondor,
how you classify different kinds of job.
They call it different universe.
And then the executable that is going to start
is this thing that I put on the submit node,
or on the log-in node, VM77.
So this is the wrapper script I mentioned
that is trying to coordinate,
try to start, try to launch an MPI processes.
And then the argument of this executable
has two different parts.
So there's actually two scripts.
The first script is to set up the environment.
The second script is to really run the MPI application itself,
which we are going to see soon.
And then we request machine count,
which is the number of nodes you have.
And per node, you want to have 16 CPUs.
And by the way, this is logical cores.
So this corresponds to 16 logical cores here.
And this is also like other things that, for example,
you tell them the jobs to transfer these files
to the worker nodes beforehand, before you start the job.
So this is the outline of what you are going to start your job.
So within that job configuration file,
we said that there's two different scripts.
So one of them is called env.sh, which set up the environment.
So let's see what it is.
And actually, we can actually do something like this.
So this is the env.sh on the right-hand side.
So you basically see that there was a file when I was--
let's see, let me switch to there.
When I was in this job configuration file,
I tell it to transfer this file to the worker nodes.
And over here, this script that starts
to prepare your environment is trying
to first uncompress your tarball to a certain location, which
is this line over here.
And then it's going to so-called activate your environment.
So after this line, now you have access to some executable.
For example, for the one that I'm providing here,
you have Python, you have mpirun.
So when you run this, like, which Python, which mpirun,
you're going to see the feedback there,
like where the Python is in that path.
And then the next part is once this environment is set up,
when your job actually starts, what it's trying to do
is to use this bash function that I provide
to set up your so-called OpenMPI host first.
And then you just, with one single line,
you are starting--
starting the MPI application like this, MPI run,
and then you specify a host by this environment variable that's
prepared by this bash function I provided.
And then Python-- in this case, it's very simple.
It's just doing some sort of hardware
trying to test your job.
So over here, probably, I can try to demonstrate it.
So I just submitted a job.
So you can see if you use this-- so if you follow through
my example over there, you will have more detailed explanation
of what this is doing.
But basically, the job has been submitted,
and it's currently still in the queue,
so nothing is happening.
And when I run this so-called tail command,
it's trying to aggressively print out the things as soon
as it starts to come to life.
So right now, the job just started,
and you can see mpi-0.out, meaning
that the first physical node is trying
to run the env.sh script over there,
and it's starting to unarchive your environment.
And also, the first process of the second physical node
is also doing the same thing.
And you wait like a minute or like 50 seconds,
and then it will tell you that, OK,
Python is now available at this path.
So right now, the job has started.
So let's go back a little bit to see what happened.
So the Python is available over here.
mpirun is available over there.
And then this mpi.log file is telling you
that these processes are actually
launched over there already.
So these different files are telling you different things.
So mpi-0.out is really the standard output
from your MPI application.
Now it's telling you, hello world, I'm from--
which processor on which node etc and then it ends so mpi.log which is what HTCondor is telling you
tells you that okay these nodes exit exceeded over there and is terminated and over here is that
job terminates on its own accord so there's no like failure over there so that's the end now i can just
stop this process. So this is a successful live demo of the MPI application. So any questions so
far? I would have one. Yes. Oh, sorry. Andrew was first. Yeah. I mean, so I lost the forest for the
trees a bit. I'm suddenly confused about communication within a node versus between
nodes. Is MPI just agnostic to that, the way it's running? Or is MPI only being used between
nodes, or I'm not sure if even my questions are
making a lot of sense.
OK, so I will try to explain it.
So let me go back to the slide.
I know this.
Can we know whether we're communicating only quickly
between-- within a single node or slowly between nodes, right?
Yeah, I need to explain a little more in the terminal.
So let me go back to this job configuration file.
Are you seeing the file?
Yeah.
Yeah, you're seeing the file.
So this is the job configuration file.
So yeah, I didn't really explain it over here.
But when we say machine counts here,
it's a little bit confusing because we are not actually
requesting number of machines.
So machine count here is more like MPI processes.
So when I request machine count equals to 2,
it means that I'm specifically launching two MPI processes.
And then when I say request CPU equals 16, it means
set per like MPI or HTCondor processors, they have 16 logical CPUs there. So one thing I was trying to
explain is those two so-called machines can actually land in the same physical node. This is part of the
confusion and sometimes can lead to some very confusing errors that you're seeing when you are
trying to debug some program. But this is something to bear in mind. Now, once we understood that,
so if you use my wrapper script to launch MPI processors, the amount of MPI processors you
have is exactly two. If you read through the documentation in that script, there's
also another mode you can launch MPI processes, which I'm not talking about here. So we only have
two MPI processes. When you are seeing the so-called like how well from the process one
one out of two there is the first MPI process.
So there's no like thread level parallelism
so far in the thing I demonstrated.
So it means that you are free to do
like hybrid kind of parallelism.
You can use MPI, sometimes called MPI plus X,
where X is often OpenMP,
if you're not doing something fancy.
So MPI plus OpenMP there, right?
So now you have two MPI processes
and within each process, you can have multi-threading.
So for example, if you use TOAST mapmaking,
by default is already using OpenMP.
So there's some sort of environmental variable,
which is also set by my wrapper script
that tells TOAST how many threads they should use
within an MPI process.
Yeah, so in short, to answer that question,
there's no like multi-threading communication
that I have shown in that demonstration.
So it's all like different MPI processes
trying to do and how well and nothing else.
- I guess I was kind of asking the opposite question.
Should you therefore assume
that all MPI communication is slow
because it might be on different nodes
and because this is only HTC and not HPC,
that can be very slow?
- You can say so, yes, you can say so.
And it's kind of like my recommended way of setting, right?
So if you are requesting like two so-called machines
that so happen in one physical node,
then the way I would do is I better just ask
for one single machine with more number of CPUs,
such that it's guaranteed to land within a physical node,
such that communication is fast
because at most it's going across the socket,
but not over the...
the network, which by the way have other complications
because I mentioned that it's 10 GbE,
but like which switch are they landing on?
And it is because it's like of the grid,
like infrastructure is not optimized in that way.
So the network topology can be complicated
and there's no guarantee that both machines
will land within the switch.
So I would say like try to avoid this, called,
like inter-network communication
as best as you can.
But if you need the memory, just like the message
I was saying there.
So usually, MPI is needed because you need more memory
than the one that is available in one single node.
If you need more memory, then of course, you
need to parallelize it there carefully.
OK, any other questions?
So I do--
All right.
Yeah, I would have one. Oh, sorry. Yes, yes, yes. Just very practical. So you told us that there was a tarball that contains sort of the programming environment. I wasn't very sure what that means. So concretely, I would like to know if you have sort of the classical programming environment, let's say at NERSC where you have your whatever you need, I mean, your maybe conda environment, maybe modules, so on.
And then you have also scripts that you want to launch from a certain directory. Is that all comprised in that tarball? Or are your scripts still in your directory?
And then the second related question, if you have sort of large input files, would they comprise a third set of inputs that you specify to kind of import from somewhere else?
So what are the three? What is the tarball doing? And where do you get input files from that are large, I would say?
Well, how do you tell to your to your launcher?
Yeah, so yeah, that was a very simple demo that I didn't touch these issues. And in a certain sense, these issues, so in terms of so I actually have written down, so that's what I'm showing in the screen right now.
So there's like software deployment, and even there's another page over here, reading and writing data, right?
So you can think of when you are preparing a node to do something, you need to read.
You need to read from a directory to load the software. So this is a software deployment over here, and also input data, output data.
So this is more complicated than what you would be doing at NERSC.
Basically, because at NERSC, everything is in a file system already. They are mounted. They are mounted in your login node.
They are also mounted on the compute node, and they are totally transparent that you don't need to.
So the only thing you need to copy is a file path, and you don't need to worry if this file will exist on the compute node.
But at a grid system here, each node, the so-called compute nodes or worker nodes over there, they are like in a blank state.
They have, they doesn't have your home, for example.
It doesn't, it doesn't see the kind of things you write at your login nodes, the vm77 there.
So the software environment you prepare over there will not work.
So that's why we need to think about how software can be deployed, which is the centre of this page.
And there are basically at least three different methods, right?
You can basically ignore the last method containers here is not supported by us for, you might say, a deficiency in HTCondor.
So the two like really de-facto standards to deploy software at a grid computing like this is either tarball or CVMFS.
And CVMFS is right now not
set up yet for our so-called virtual organization.
So it means that right now, the tarball method
is the only way we can do it.
And there's a software package that I
provide that includes all the dependencies and sotodlib.
So it also includes SPT-3G, I think.
And also, what else?
Like, I put it there.
But of course, it doesn't include everything.
For example, currently, the so-called NaMaster
is not there yet.
So I also, in one page--
actually, in this page of tarball method, in the end of that page,
I mentioned how you can tailor your tarball, basically.
So not everything I provided--
I mean, whatever you need might not be provided by me.
So you need to add something, or you
want to have a newer version of TOAST, let's say.
And this is sort of an internal--
I'm not going to do that.
It's not very clear yet right now.
But it will improve over time.
And by the way, yeah, I should address that.
So when I say--
OK, so the next question is, do we
have time to make one more demo, which
will answer your next question, like doing I/O to the nodes?
So do you want to have one more demo?
Maybe I can just read through them.
Yeah, I think probably don't do the other demo Kolen, because we only
have sort of minutes till kind of hard deadline.
Yeah, so we will just--
which is also more safe to me.
The next demo is more complicated.
So now we are going to talk about how to read and write data there.
So basically, there's-- let's see.
By the way, this session is very long, because in a certain sense,
it's more complicated.
So there's three big sessions.
If you look at the table of contents on the left-hand side,
So one is just transferring file from via the so-called ClassAd, the job
configuration file, using HTCondor. So this is the simplest one but also the
most limited in terms of capability and is something we are not recommending.
Basically the submit node is very small. We only have 200 GB of this space
and if you want to use this basically you need to assume everything is on
your submit node either like reading from or transmitting to. So this is not a
recommended method so I will skip it here but but the example is there you
can try. So this is the next method the so-called grid storage system is the
de-facto standard that on the grid they were committed to do it this way. So this
is the main thing and lastly there's something we are not going to mention
today is the Librarian. So we I think we all know that we are going to load some
the TOD from librarian, but it's more a special case of loading data. So I'm not going to talk
about it right now. And also it hasn't been set up yet. So in terms of grid storage system,
so I, this documentation goes through it in detail. So you see, there's a lot of sub pages
within this. Many of them is related to setting things up. But fortunately, if you're using vm77
is easier. So the thing you need to do is with the setup that we already
created, let's see this part. So the, the thing that you need to do is starting from here. So
you, you need your grid certificate. You put it in a specific location, which is this path,
basically. And then once you have that certificate over there,
the tool set you're going to use down there is going to authenticate your jobs via your user
certificate, basically. So you need to create something periodically too. So what happens is
when you run this command, it's actually trying to create something called the AC attribute
certificate, which is temporary. Your user certificate is a permanent thing that ties to you.
And using your certificate, this will create something temporary that you can use to submit
a job. And once you run this command, this AC is going to be valid for one week.
So it means that every week you need to run it once, like every Monday morning you're trying
to run it once. And it will tell you this sort of a demo that if you run this command,
this is what you're going to see. And it will say a certain proxy is created over here,
and the proxy is going to be valid until that time.
Once you have that file, you can then start your job.
So this, I think you can, can't ignore that.
So this is a job demo.
So basically, once you have your AC certificate,
then you have a job conversion file like this,
which transfer your AC to the worker nodes.
And then in your job script on the worker nodes,
there's one single command you need to set,
export this environment variable.
And after this, your worker nodes will be able to read
and write to and from the so-called grid storage system.
So this script is basically a demonstration on that.
You can try to use a special kind of list command,
so-called gfal-ls,
to see what is available within your grid storage system.
And then you can try to, over this line,
you, let's see, I try to make a directory first.
And then in the next line,
I try to remove that directory again,
and I create some file and you use this command
to copy that file to your grid storage system.
So this is the example of how you can
interact with the grid storage system.
And, and for
the bulk of the data that we are going to have,
like about a hundred terabytes right now,
is going to be only accessible using this way.
And in the long run, by the way,
there will be another system
which will make something like this easier.
But in the interim, let's say for the next months,
we need to do something like this.
Okay, so probably, I have more documentation over there,
which you can try to read on your own,
but I will go back to the presentation.
So this was the live demo.
Any other questions so far?
Okay, so this last big session is like to be explored,
like the how over there,
which is like the future aspects of things.
So this slide basically mentioned
how to design a workflow for a system, right?
So I tried to mention something about effectiveness.
You can think about it in two different ways.
One is amount of compute time,
so you can say how much amount of fair share
you have used up from your system,
like allocation, like NERSC hours, something like that.
And also in terms of the turnaround time,
like how fast, once I submit a job,
how quickly can I get the result, right?
This is what I define to be effectiveness of a workflow.
And for more capable system,
like it is more lenient on sub-optimality, right?
So if you write a workflow that is not very optimal,
but you know that it works,
a more capable system is more lenient to that.
Meaning that if you want to adapt a workflow from NERSC
to the SO:UK Data Centre,
you might see bottlenecks that you haven't been thinking about.
So one example I gave here is that,
for example, in mapmaking,
so let's say I have 500 hours of observation.
Each map making is going to inject one single hour observation
and provide you one single map, right?
So now I have 5,000 jobs.
You can launch a very wide MPI application with 5,000 processes
and it will give you the answer in a very short amount of time,
let's say three minutes, right?
But this is not a very good use of NERSC resource.
The reason is now you have a couple of other problems.
So it's load-balancing.
So do all the mapmaking finish at the same time
and also turnaround time,
because you are asking for a very wide job.
How soon will the scheduler be able
to put your job in the queue. So when you adapt that kind of workflow to Blackett, it will be
even worse. So if you try to start 5,000 processes, I don't know how long would it be able to start.
But if you bring it down into 500 different jobs, then basically most of them will start instantly.
So in this aspect, if you tune your workflow specifically for Blackett, you can actually
have faster turnaround time in at least in some situation because at NERSC if you if you try you
know something called debug partition that's not supposed to be in production if you put it in
production queue the typical turnaround time is like at least three days so if you want to run
a map that will finish in three minutes it will be most probably going very faster over here so the
recommend workflow then is to launch more small jobs right so you just launch each job is an
atomic job by atomic i mean that that single job has it is not going to communicate with any other
job like kind of like the mapmaking I was talking about you're not doing some you're only doing
naive mapmaking each process are not communicating so each of them are reading their own TOD and
writing their own maps so each of them is one atomic job so if you try to factor that out
it will actually help you to go through the uh the queue faster and
but then you now have a complication because now you need to need to launch tens of thousands of
jobs right how do you handle it so there's something workflow manager is going to help you
and i can't talk about more details about here but like i kind of discourage which is also
what NERSC is discouraging about rolling your own workflow manager which basically because
workflow managers are trying to solve these issues right so we can think about like you see job
managers agnostics—do your workflow manager assumes the presence of SLURM5 for example
and how job dependencies are handled so let's say if you after you start map making you also
have other things that depends on the finished maps right how how would those be launched later
on do you need to like kind of babysit your job and launch it manually, how failed jobs are handled,
like do you need to check for corrupted maps from individual outputs etc so
One caveat I want to mention here is, while filter-bin mapmaking is well-suited for MapReduce
product, but we need to think about how you treat the data from each map. Map in the MapReduce
sense, not necessarily the CMB maps. So meaning that each function, once you have the product,
do you write it on disk or otherwise? So the reason is, if you write it on disk, then you need
you need to deal with two extra problems-- the data explosion.
You have a lot of intermediate data you need to write down.
And also, you are congesting the interconnects.
So basically, because once you start to write a file,
you actually need to go through the interconnect
and write it somewhere else.
So now the network condition, the link of the network
becomes more apparent.
So yeah, this is the thing I want to mention.
And last slide, sorry.
Yeah, this is the last slide, documentation.
So I already mentioned that the documentation site is
available here.
I will also copy and paste in the chat there very soon.
And how you can use the documentation.
So I documented a few ways.
So of course, you can search it.
So if you just click this, so basically everything
is indexed and searchable over there, which is very useful.
And you can also download different formats,
like PDF, EPUB, man page, single page HTML.
And how would those formats be useful?
you can actually try to fit in that to a large language model, right?
So you can now start to chat with your model.
For example, if you inject your single-page HTML to ChatGPT,
and you can try to summarize what is this documentation,
what do we have in the documentation,
and then start chatting with the document.
So tell me to run an MPI application.
According to the documentation, how should I do it?
Something like that.
And whatever questions you have, you can try asking that.
But of course, sometimes it might hallucinate your answer.
But anyway, so these are some of the ways you can use the documentation.
And you can also start to collaborate and discuss issues here, right?
So there's a GitHub repository there.
You can raise issues, and we already have a lot of them here.
And also discuss things over here, GitHub discussions, et cetera.
Or even to contribute, right?
So at the page of onboarding, I mentioned only the UK certificate.
So if you have a UK certificate, you can use the URL over there.
But if you know other ones, you can try to add the document, submit a pull request.
Okay, so that's all for the introduction of our data centre.
Thank you.
Cool. That's brilliant. Thanks, Kolen.
Right, we're close to five o'clock.
Does anyone have any quick questions for Kolen?
Okay, so we're close to five o'clock.
Thank you.