CSD Related Links: Difference between revisions

From IHRIS Wiki
Line 83: Line 83:
*Loading the CSD.xml document in zorba's memory store required 2.5mb.
*Loading the CSD.xml document in zorba's memory store required 2.5mb.
*Results on  m1.large Amazon EC2 instance
*Results on  m1.large Amazon EC2 instance
Socket AF_UNIX listening on local host
<pre>
<pre>
httperf --server localhost --uri /CSD/zorba_manager.php/search/search_by_nid.xq?nid=1197580084757020 --rate 100 --num-conn 2000    --num-call 1 --timeout 1
$ httperf --hog --server localhost --uri /CSD/zorba_manager_uds.php/search/search_by_nid.xq?nid=1197580084757020         --num-conn 100
httperf --timeout=1 --client=0/1 --server=localhost --port=80 --uri=/CSD/zorba_manager.php/search/search_by_nid.xq?nid=1197580084757020 --rate=100 --send-buffer=4096 --recv-buffer=16384 --num-conns=2000 --num-calls=1
httperf --hog --client=0/1 --server=localhost --port=80 --uri=/CSD/zorba_manager_uds.php/search/search_by_nid.xq?nid=1197580084757020 --send-buffer=4096 --recv-buffer=16384 --num-conns=100 --num-calls=1
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE
Maximum connect burst length: 1
Maximum connect burst length: 1


Total: connections 2000 requests 65 replies 0 test-duration 21.417 s
Total: connections 100 requests 100 replies 100 test-duration 2.403 s


Connection rate: 93.4 conn/s (10.7 ms/conn, <=112 concurrent connections)
Connection rate: 41.6 conn/s (24.0 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 0.0 avg 0.0 max 0.0 median 0.0 stddev 0.0
Connection time [ms]: min 21.3 avg 24.0 max 31.6 median 23.5 stddev 1.5
Connection time [ms]: connect 690.4
Connection time [ms]: connect 0.1
Connection length [replies/conn]: 0.000
Connection length [replies/conn]: 1.000


Request rate: 3.0 req/s (329.5 ms/req)
Request rate: 41.6 req/s (24.0 ms/req)
Request size [B]: 132.0
 
Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (0 samples)
Reply time [ms]: response 23.9 transfer 0.0
Reply size [B]: header 193.0 content 171.0 footer 0.0 (total 364.0)
Reply status: 1xx=0 2xx=100 3xx=0 4xx=0 5xx=0
 
CPU time [s]: user 0.49 system 1.70 (user 20.5% system 70.7% total 91.2%)
Net I/O: 20.2 KB/s (0.2*10^6 bps)
 
Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
</pre>
Socket AF_INET listening on local host
<pre>
httperf --hog --server localhost --uri /CSD/zorba_manager.php/search/search_by_nid.xq?nid=1197580084757020        --num-conn 100
httperf --hog --client=0/1 --server=localhost --port=80 --uri=/CSD/zorba_manager.php/search/search_by_nid.xq?nid=1197580084757020 --send-buffer=4096 --recv-buffer=16384 --num-conns=100 --num-calls=1
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE
Maximum connect burst length: 1
 
Total: connections 100 requests 100 replies 100 test-duration 1.623 s
 
Connection rate: 61.6 conn/s (16.2 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 14.1 avg 16.2 max 20.3 median 16.5 stddev 0.8
Connection time [ms]: connect 0.1
Connection length [replies/conn]: 1.000
 
Request rate: 61.6 req/s (16.2 ms/req)
Request size [B]: 128.0
Request size [B]: 128.0


Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (4 samples)
Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (0 samples)
Reply time [ms]: response 0.0 transfer 0.0
Reply time [ms]: response 16.1 transfer 0.0
Reply size [B]: header 0.0 content 0.0 footer 0.0 (total 0.0)
Reply size [B]: header 192.0 content 34.0 footer 0.0 (total 226.0)
Reply status: 1xx=0 2xx=0 3xx=0 4xx=0 5xx=0
Reply status: 1xx=0 2xx=100 3xx=0 4xx=0 5xx=0


CPU time [s]: user 1.53 system 19.82 (user 7.1% system 92.5% total 99.7%)
CPU time [s]: user 0.45 system 1.08 (user 27.6% system 66.6% total 94.2%)
Net I/O: 0.4 KB/s (0.0*10^6 bps)
Net I/O: 21.3 KB/s (0.2*10^6 bps)


Errors: total 2000 client-timo 2000 socket-timo 0 connrefused 0 connreset 0
Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
</pre>
</pre>

Revision as of 09:05, 29 June 2013

CSD Links

  • CSD
  • XQuery Engines
  • XQuery Engine Comparison
    • Evaluation Criteria: Programming Language, Source Code Last Updated, Source Documentation, XQuery verison, XQuery Update Facility support, Data storage, Index, Client libraries (PHP, Python, Java,etc.), Mailing List/User Forum last activity

Potential Issues for Public Comment

  • how to map hcIdentifier used in HPD to II ?
  • no date of birth in persons
  • do we need Org/Facility Contact to be required?
  • orgType, orgTypeDescription, orgPrimaryName are all required but perhaps should not be. similarly for facility.
  • minoccurs on orglanguage is not set, but is set for facilityLanguage
  • no place to add other identifiers for facility
  • should creation date be required?
  • facility can be in multiple facilityorgid? how to represent one facility in multiple hierarchies that are not necc. mapped to services
  • is a lastmodified time/create time for CSD doc needed?
  • CareServicesDiscoveryQuery/XQueryFunction has no ability to set parameters (e.g. nid). Options would include:
    • for un-ordered named variables something we define such as 
      <XQueryFunction name="loopup_by_nid.xq"><value name="nid">123</value></XqueryFUnciton>
    • XML-RPC for ordered unnamed variables
    • some other structured XML that knows about xquery data types (note: XQuery shares the same data types as XML Schema 1.0 , XSD).
  • CareServicesDiscoveryQuery/XQueryResult/queryResult may not be valid XML.. Should it always be a CDATA?

Zorba Implementation Notes

installed subversion according to https://help.ubuntu.com/community/Subversion

  • subversion installed under /home/svn/CSD with autoversioning for PUT files
  • webdav under http://csd.ihris.org/svn/CSD
    • user: csd pass: ASK CARL FOR IT. It is also in /home/ubuntu/CSD/local/config.php
  • to check out: svn co http://csd.ihris.org/svn/CSD CSD --username csd
    • currently checked out to svn sub-directory of dropbox folder
  • checked out the repo under /home/ubuntu:
  • in svn repository, the www directory should be linked to under /var/www:
    • sudo ln -s /home/ubuntu/CSD/www /var/www/CSD
  • webdav php client is installed with "sudo apt-get install php-sabredav"
  • zorba does not install exactly according to the instructions found online. In particular make sure you:
    • do "sudo apt-get install swig" before running cmake
    • do "-D CMAKE_INSTALL_PREFIX=/usr/local .. " in the build directory

CareServicesConsumerQuery

Submit a POST request to zorba_manager.php with your document. Some sample documents are found in test_csdq

You can test this with, for example:

 curl -X POST -d @/home/ubuntu/CSD/test_csdq/stats_named.xq.xml http://csd.ihris.org/CSD/zorba_manager.php

Searches

Named/Canned Xquery searches are made available under queries/search via the web dav server. If there is a file queries/search/blah/blah.xq, you can access it as http://csd.ihris.org/CSD/zorba_manager.php/search/blah/blah.xq. The dynamic context of the search is the published CSD document at time of the zorba server initialization. For the time being, and GET variables are passed on to the query. This is wide open at the moment and we need to lock it down.

Publishing

Named/Canned Xquery use to publish CSD documents are made available under queries/publish via the web dav server. If there is a file queries/publish/blah/blah.xq, you can access it as http://csd.ihris.org/CSD/zorba_manager.php/publish/blah/blah.xq. Each document under docs, with the exclusion of CSD.xml, is available to the xquery. If the resulting document passes validation via the .xsd it is published to the web dav server.

Currently there is one publishing script queries/publish/simple_merge.xq.


Currently, there is no way to tell the zorba server to reload/refresh the CSD source document after publication. You need to login to the server and restart the "zorba_server/init.php" process manually.

Updating

Named/Canned Xquery use to publish CSD documents are made available under queries/publish via the web dav server. If there is a file queries/publish/blah/blah.xq, you can access it as http://csd.ihris.org/CSD/zorba_manager.php/update/blah/blah.xq. Each document under docs is available to the xquery and "CSD.xml" is set to be the default context. If the resulting document passes validation via the .xsd it is published to the web dav server.

There are no update scripts as of yet. The intention is that scripts here are used to update the currently published CSD.xml, for example with an update of facilities. How this update occurs is defined by the update script. An example workflow would be:

  • XML data exported from FRED-API compliant FR with updates since a given time.
  • This data is posted to webdav server under the docs/ directory as, for example, "fr_updates.xml"
  • There is an xquery script, for example "fr_update.xq", under "queries/update/" that would:
    • scan fr_updates.xml for any new facilities and add them to the CSD.xml, which is the default context
    • scan fr_updates.xml for any existing facilities and either delete/replace the entire <facility> node, or overwrite any values contained within it as necessary.

Profiling

Currently there are no indices are the zorba server's data store.

  • It can process about 3 NID look-ups per second through the web interface on localhost
  • Actually executing the query takes, for example, 0.0078308582305908 seconds
  • Loading the CSD.xml document in zorba's memory store required 2.5mb.
  • Results on m1.large Amazon EC2 instance

Socket AF_UNIX listening on local host 

$ httperf --hog --server localhost --uri /CSD/zorba_manager_uds.php/search/search_by_nid.xq?nid=1197580084757020         --num-conn 100 
httperf --hog --client=0/1 --server=localhost --port=80 --uri=/CSD/zorba_manager_uds.php/search/search_by_nid.xq?nid=1197580084757020 --send-buffer=4096 --recv-buffer=16384 --num-conns=100 --num-calls=1
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE
Maximum connect burst length: 1

Total: connections 100 requests 100 replies 100 test-duration 2.403 s

Connection rate: 41.6 conn/s (24.0 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 21.3 avg 24.0 max 31.6 median 23.5 stddev 1.5
Connection time [ms]: connect 0.1
Connection length [replies/conn]: 1.000

Request rate: 41.6 req/s (24.0 ms/req)
Request size [B]: 132.0

Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (0 samples)
Reply time [ms]: response 23.9 transfer 0.0
Reply size [B]: header 193.0 content 171.0 footer 0.0 (total 364.0)
Reply status: 1xx=0 2xx=100 3xx=0 4xx=0 5xx=0

CPU time [s]: user 0.49 system 1.70 (user 20.5% system 70.7% total 91.2%)
Net I/O: 20.2 KB/s (0.2*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

Socket AF_INET listening on local host 

httperf --hog --server localhost --uri /CSD/zorba_manager.php/search/search_by_nid.xq?nid=1197580084757020         --num-conn 100 
httperf --hog --client=0/1 --server=localhost --port=80 --uri=/CSD/zorba_manager.php/search/search_by_nid.xq?nid=1197580084757020 --send-buffer=4096 --recv-buffer=16384 --num-conns=100 --num-calls=1
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE
Maximum connect burst length: 1

Total: connections 100 requests 100 replies 100 test-duration 1.623 s

Connection rate: 61.6 conn/s (16.2 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 14.1 avg 16.2 max 20.3 median 16.5 stddev 0.8
Connection time [ms]: connect 0.1
Connection length [replies/conn]: 1.000

Request rate: 61.6 req/s (16.2 ms/req)
Request size [B]: 128.0

Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (0 samples)
Reply time [ms]: response 16.1 transfer 0.0
Reply size [B]: header 192.0 content 34.0 footer 0.0 (total 226.0)
Reply status: 1xx=0 2xx=100 3xx=0 4xx=0 5xx=0

CPU time [s]: user 0.45 system 1.08 (user 27.6% system 66.6% total 94.2%)
Net I/O: 21.3 KB/s (0.2*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
Retrieved from "https://wiki.ihris.org/mediawiki/index.php?title=CSD_Related_Links&oldid=48687"