PDF《Data Streaming》

? Paul Scherrer Institute - simon.

[email protected] Paper ID: WED3O06 Data Streaming Efficient Handling of Large and Small (Detector) Data at the Paul Scherrer Institute S. Ebner1, H. Billich1, H. Brands1, Ezequiel Panepucci1, L. Sala1

1 Paul Scherrer Institut (PSI), CH-5232 Villigen PSI, Switzerland Paper ID: WED3O06 ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 The Paul Scherrer Institute, PSI, is the largest research centre for natural and engineering sciences within Switzerland http://www.psi.ch SwissFEL Swiss Free Electron Laser SLS Swiss Light Source SINQ Swiss Spallation Neutron Source HIPA The high-intensity proton accelerator ... ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Outline Motivation Objectives Design Lessons Learned Outlook Questions ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Motivation ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Motivation Detector Viewer File ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Motivation Detector Viewer File Detector X File Viewer ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Motivation Detector Viewer File Detector X File Viewer File Formats ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Motivation Detector Viewer File ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Motivation Detector Viewer File Analysis ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Motivation Detector Viewer File Analysis Processing ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Motivation Detector Viewer File Analysis Processing I/O Limited ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Motivation Detector Viewer HDF5 File ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Motivation Detector Viewer HDF5 File Delay / Concurrency ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Motivation … ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Objectives ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Objectives Minimize Complexity Code Design Dependencies Conflicts ... ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Objectives Maximize Flexibility Programming Language Design Simplicity Decoupling ... Exchangeability (Code) Reuse Parallelization Scaleability Maintainability Stability ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design Principles ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design FRAMEWORKS SUCK Principles ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design Data on Disks is DEAD FRAMEWORKS SUCK Principles ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design Data on Disks is DEAD FRAMEWORKS SUCK Principles Separation of Concerns Control/Data Data taking, viewing, analysis, persistence, … ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design Data on Disks is DEAD FRAMEWORKS SUCK Principles Standardisation/ APIs on the Network Separation of Concerns Control/Data Data taking, viewing, analysis, persistence, … ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design Detector Persistence ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design Detector Persistence ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design Detector Persistence ZMQ ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design Detector Persistence ZMQ PUSH/PULL PUB/SUB Delivery Scheme: ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design Detector Persistence ZMQ HWM (HighWaterMark) HWM PUSH/PULL PUB/SUB Delivery Scheme: ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design Detector Persistence ZMQ HWM (HighWaterMark) HWM PUSH/PULL PUB/SUB Delivery Scheme: Control Control ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design - Message Sub Message ... Main Header (JSON) 0..* required {"htype":"array-1.0", "shape":[10,20], "type":"uint16", "frame":0} htype defines content of main header as well as the structure of the whole message (sub messages) Sub Message(s) can be binary or JSON ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design Detector Persistence ZMQ HWM (HighWaterMark) HWM PUSH/PULL PUB/SUB Delivery Scheme: Control Control ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design - Receiver Example import zmq context = zmq.Context.instance() sock = context.socket(zmq.PULL) sock.connect('tcp://hostname:9999') while True: header = sock.recv() print header while socket.getsockopt(zmq.RCVMORE) data = sock.recv() ? Paul Scherrer Institute - [email protected] Paper ID: WED3O06 Design - Persistence Example import zmq context = zmq.Context.instance() sock = context.socket(zmq.PULL) sock.connect('tcp://hostname:9999') counter =