CMS: Transfer files from EOS to CTA

[psaiz]

Our CMS colleagues have identified three sets of files that could be migrated to cold storage (thanks @jmhogan !):

• 7 TeV AODSIM
• 2016 MINIAODSIM
• 8 TeV AODSIM

They should be copied to tape first

[psaiz]

Let's document the process, with the first link (the 7 TeV):

1. Identifying the records to copy:

curl 'https://opendata-dev.cern.ch/search?q=&f=experiment%3ACMS&f=file_type%3Aaodsim&f=collision_energy%3A7TeV&l=list&order=desc&p=1&s=10000&sort=mostrecent'  | jq '.hits.hits[].metadata.recid ' | paste -sd ' ' - | tr -d '"'
1681 809 808 807 806 805 804 803 802 801 800 799  ...

2. Checking the size of those records (from the worker pod):

cernopendata  cold list 1681 809 808 807 806 805 804 ...
...
Summary: 147140 files (273.8 TiB), with 147140 hot copies (273.8 TiB) and 0 cold copies (0 bytes) 

3. Sending the command to copy to tape, in batches

cernopendata  cold archive 1681 809 808 807 806 805 

And that should do it. It can be monitored here

It can also be monitored here

[psaiz]

There are more entries than I expected. I'm changing the approach slightly, with a queue of records to transfer and then ensuring than in FTS there are around 1000 transfers at a given time.

The total amount of files to transfer:

• 7 Tev: 482 records, 147k files, (270 TiB)
• 2016: 21584 records, 363k files (469 TiB)
• 8 Tev: 377, 275k files (666 TiB)

[jmhogan]

Thanks @psaiz

I'm curious about the volumes. These are all quite a bit smaller than what we had counted up.
For instance, du reports that /eos/opendata/cms/MonteCarlo2012/ is 844T
And it reports that /eos/opendata/cms/mc/RunIISummer20UL16MiniAODv2 is 738T