sobota 12. října 2019

Is file_get_contents & file_put_contents reliable or can lead to loss of data? Benchmark results

I was wondering what happens if multiple scripts are sharing same file. I uploaded the test on remote server, where they use HDD to store data. There were 7 tests total, but the family of 6 are compatible.
I have 7 files of different size which I uploaded to server and the test. It is loop which reads and writes data from the files.
There is 50 microseconds delay in the loop. The loop repeats 50x.
I measure the time needed to perform every circle.

The differences in the tests (T):

Using file_get_contents/file_put_contents
T2 - SOURCE <> TARGET - reads data from original file, writes data do different (new) file
T3 - SOURCE = TARGET - 1. copies data from original file to target; 2. reads source data -> writes data; 3. the point 3 is repeated: i.e I read the data which I have written. This test uses same file to write data.
T4 - SOURCE = TARGET - I repeated the same test as in T3 getting shorted times.
Using fopen, flock, fread, flock, fclose, fopen, flock, fopen, fwrite, fflush, fclock, fclose ... This is complicated code, but here I have tested the fflush. I also use clearstatcache, stat and touch and clearstatcache, filesize. To check validity. The tests T5 - T7 were less reliable than T2-T4 because sometimes the write operation failed. I tested the file size and when it was not correct, I copied (restored) the file back from original file.

T5: (fflush) SOURCE = TARGET
T6: (fflush) SOURCE <> TARGET
T7: (fflush) SOURCE <> TARGET + I have removed the 50 microseconds delay from the loop (It seems like the validity/reliability is worse when there is a delay).
I made 4 requests from 4 different browsers - so every test have 4 sets of data (7*50*4 values total).

Now I have collected all data, created tables and diagrams. This is one diagram of many, showing minimal and maximal values of avarage value.

T4 yellow color and T3 green provides very small times so they are suspicious. For example T4 avarage times are these: 0,001

0.001 0.002 0.003 0.002 0.004 0.003 0.004 0.001 0.004 0.001 0.004 0.001 0.004
 
And T3 times:

0.002 0.003 0.001 0.001 0.003 0.003 0.006 0.007 0.002 0.003 0.004 0.004 0.019 0.019
 
The values of T2 seems normal, but this can be explained by the fact, that that was read from different file than was written to.

T5-T7 just show normal times as expected - the bigger the file the bigger the time needed to proccess. Fairly slow as expected from HDD and 4 scripts running at the same time.
So my question here is:

Does the results of T3-T4 mean, that the file_read_contents and file_put_contents are not reliable for this type of job? To me it looks like they simply do not read the data from file but they are copied from buffer, which means, that old data are saved, not the current data been changed by concurent script. I would welcome more information. I spent a lot of time searching for answers but did not found clear answer. I did this tests because I need proofs. You man want to use my scripts but I am not sure if can I paste here the 6 scripts?

 Now I will add just the fflush test number 7 which is most useful.



PHP 
clearstatcache();
$_DEBUG_ = false;

echo "Lock and flush tester.".time()."
";
die;

while ( time()<1570787996 )
 {
 usleep(500);
 }


function test($n, $p, $_DEBUG_){
  $sname = "$n";    // source
  $tname = "$n.txt";// target
  echo "

$n at "

.time()."
"; for ($i = 0; $i<50; $i++ ){ $start = microtime(true); clearstatcache(); // needed for filesize and touch $st = stat("$sname"); $original_size = $st['size']; if ( $_DEBUG_ ) echo "; 1) prevAccess by ".$st['mtime']." fsize ".$st['size']."; "; $fsize = filesize($sname); if ( $original_size <> $fsize ) die("; fsize total FAILTURE; "); if ($fsize === 0) echo "! The fsize is 0: stat(): ".$st['size']." ;"; else { // READ OPERATION AND LOCK FOR SHARE $locked = false; for ($c = 0; !$locked; $c++): if ( $c > 400) break; $fp = fopen($sname, "r"); $locked = flock($fp, LOCK_SH); if ($locked) break; else { echo "failed to get LOCK_SH;
"
; usleep(5000); } endfor; $s = fread($fp, $fsize ); $success = flock($fp, LOCK_UN); if ( $success === false ) die("; r flock release failed; "); $success = fclose($fp); if ( $success === false ) die("; fclose failed; "); // 10 - data loaded , $p - browser if ( $success ) { $result = touch("$sname",strlen($s),$p); if ( $_DEBUG_ ) echo "; TOUCH: $result;"; } else die("fclose FAIL."); if ( strlen($s)<60 ) echo "*$s LENGTH:".strlen($s)."
"
; } clearstatcache(); $st = stat("$tname"); if ( $_DEBUG_ ) echo "; 2) prevAccess by ".$st['mtime']." fsize is ".$fsize."; "; // WRITE OPERATION WITH LOC_EX $fp = fopen($tname, "w"); $locked = false; $locked = flock($fp, LOCK_EX); if ( $locked ) { // acquire an exclusive lock $success = fwrite($fp, $s); if ( $success === false) echo "; w FAILED;"; else if ( $_DEBUG_ ) echo " $success B written; "; $success = fflush($fp);// flush output before releasing the lock if ( $success === false ) echo "; flush FAILED; "; $success = flock($fp, LOCK_UN); // release the lock if ( $success === false ) echo "; release FAILED; "; $success = fclose($fp); if ( $success === false ) echo "; fclose FAILED; "; clearstatcache(); // needed for filesize and touch $fsize = filesize($tname); if ($original_size>$fsize) { echo "; WRITE FAILED, restoring;"; $original_fname = "$n"; $result = copy($original_fname, $tname); if ($result == false ) die(" TOTAL FAILTURE: copy failed."); else echo " RESTORED;"; } else { if ($fsize === 0) echo "! THE FILE WAS NOT WRITTEN: data length: ".strlen($s)." fsize: $fsize RESOURCE: $fp
"
; if ( $success ) touch("$tname",$fsize,$p); } } else { echo "Couldn't get the lock!"; } $time_elapsed_secs = microtime(true) - $start; if ( $time_elapsed_secs === 0 ) echo " FAILED "; echo "time: $time_elapsed_secs s
"
; } } switch ( $_SERVER['HTTP_USER_AGENT'] ): // FF 1: case "Mozilla/5.0 (Windows NT 5.1; rv:49.0) Gecko/20100101 Firefox/49.0": $p = 1; break; // Chrome: case "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36": $p = 2; break; // OPERA: case "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 OPR/36.0.2130.80": $p = 3; break; endswitch; copy("523","523.txt"); copy("948","948.txt"); copy("1371","1371.txt"); copy("1913","1913.txt"); copy("2701","2701.txt"); copy("4495","4495.txt"); copy("6758","6758.txt"); test("523",$p,$_DEBUG_); test("948",$p,$_DEBUG_); test("1371",$p,$_DEBUG_); test("1913",$p,$_DEBUG_); test("2701",$p,$_DEBUG_); test("4495",$p,$_DEBUG_); test("6758",$p,$_DEBUG_); die; echo "php: " . phpversion(); ?> PHP echo "php: " . phpinfo(); ?>
 
Notice: 523 means the filesize is 523 kB.


You may want to enable $DEBUG option to monitor each proccess. Note: The touch maybe do not work correctly always.
Note: This is not a request for test, this is just request for review.


Also: Please do not be confused by the yellow color curve. There are two yellow colors. The T4 yellow is almost no visible on the diagram because it has very low values.

If you want to solve this question, follow the topic here.

<50 1="" and="" by="" clearstatcache="" echo="" filesize="" for="" fsize="filesize($sname);" i="" if="" mtime="" needed="" original_size="" prevaccess="" size="" sname="" st="" start="microtime(true);" touch=""><60 br="" echo="" length:="" s="" strlen="">

Žádné komentáře:

Okomentovat

Jak zkopírovat styly jako font a odstavec v LibreOffice Write?

V dokumentu Write píšu CV. Někde uprostřed mám nadpisy a chci zkopírovat jeden nadpis v četně stylů a vložit ho jinde. Když použiju např ct...

Štítky

.profile adm AI alfa transparence AND any aplikace asociativní pole atomicity audio awk bash benchmark bezpečnost biblehub BJT boolean buffering Cache-Conrol Cloudflare code Collector Cut-off colorpicker compare cookies css CSS3 curl cut čas data loss data lost data transfer reliability datasheet datetime.strptime development dioda diody EBO Emitter Cut-off Current ETag exclude exec Expires fflock fflush ffmpeg file read file write file_get_contents file_get_contents/file_put_contents file_put_contents filter find first_install.sh flock Fly-back dioda font-face fóra fotorezistor fread functions funkce fwrite gate gate drive geolokace gradient-background grep grub grub update hebrejština history hlavičky HS html html 5 https hudba charakterizace chroot ICES IGBT img sizes img srcset impedance implementace imshow inference inrush current install jalový výkon javascript javescript jednocení seznamů js kapacita součástek koeficient zesílení komunikace se serverem konfigurace Krita KSF Last-Modified lazy caching led LEFT JOIN librosa ligatury linux list log manuál masky matplotlib Max-Age measure memory měření MFCC MFCC koeficienty mint míry modules moralizace morphologie MOSFET mount moviepy mysql náběhový proud napěťová ochrana návod nel Network Error Logging NLP not nth-child oblékání ochrana okruhy přátel OpenVINO IR formát optočlen ořezové masky OSHB otázky otázky_jazyky otázky_moralismu_řešení parsování path personifikace photorec php php 4 php 5 php 6 php 7 php 8 phpbb phpBB3 pitch PN přechody pnp pole práva profilování program prune průraz přepěťová ochrana přepolování pseudokódd PWM regulátory pydub python python3 pytorch RBE RDSon read reaktance rectifier regulace vstupního napětí relyability remount replace restore reverzní geolokace RIGHT JOIN rm role rozvržení disků pro OS linux a data databází řešení samba sdílení Sec-Fetch-Dest Sec-Fetch-Mode Sec-Fetch-Site Sec-Fetch-User Secure Shell sed Set Cookie shunt schottka skript sloupce song sort soubory soundfile spínané zdroje spínání split SQL ssh stabilizace napětí stahování stream string strojové učení stropové učení syntax T5 tabulky tepelná ztráta test text-shadow thermal runaway time timestamp tkinter tr transistor transition tranzistor tranzistory tuple tvorba otázek TVS účiník update va charakteristika Vgs video Vth vynechání adresářů vývoj while wrapovací funkce XOR zdánlivý výkon zdroj zenerka zenerovo napětí zip zip archiv zkratky zpomalení zpracování textu Žalmy