sobota 12. října 2019

Is file_get_contents & file_put_contents reliable or can lead to loss of data? Benchmark results

I was wondering what happens if multiple scripts are sharing same file. I uploaded the test on remote server, where they use HDD to store data. There were 7 tests total, but the family of 6 are compatible.
I have 7 files of different size which I uploaded to server and the test. It is loop which reads and writes data from the files.
There is 50 microseconds delay in the loop. The loop repeats 50x.
I measure the time needed to perform every circle.

The differences in the tests (T):

Using file_get_contents/file_put_contents
T2 - SOURCE <> TARGET - reads data from original file, writes data do different (new) file
T3 - SOURCE = TARGET - 1. copies data from original file to target; 2. reads source data -> writes data; 3. the point 3 is repeated: i.e I read the data which I have written. This test uses same file to write data.
T4 - SOURCE = TARGET - I repeated the same test as in T3 getting shorted times.
Using fopen, flock, fread, flock, fclose, fopen, flock, fopen, fwrite, fflush, fclock, fclose ... This is complicated code, but here I have tested the fflush. I also use clearstatcache, stat and touch and clearstatcache, filesize. To check validity. The tests T5 - T7 were less reliable than T2-T4 because sometimes the write operation failed. I tested the file size and when it was not correct, I copied (restored) the file back from original file.

T5: (fflush) SOURCE = TARGET
T6: (fflush) SOURCE <> TARGET
T7: (fflush) SOURCE <> TARGET + I have removed the 50 microseconds delay from the loop (It seems like the validity/reliability is worse when there is a delay).
I made 4 requests from 4 different browsers - so every test have 4 sets of data (7*50*4 values total).

Now I have collected all data, created tables and diagrams. This is one diagram of many, showing minimal and maximal values of avarage value.

T4 yellow color and T3 green provides very small times so they are suspicious. For example T4 avarage times are these: 0,001

0.001 0.002 0.003 0.002 0.004 0.003 0.004 0.001 0.004 0.001 0.004 0.001 0.004
 
And T3 times:

0.002 0.003 0.001 0.001 0.003 0.003 0.006 0.007 0.002 0.003 0.004 0.004 0.019 0.019
 
The values of T2 seems normal, but this can be explained by the fact, that that was read from different file than was written to.

T5-T7 just show normal times as expected - the bigger the file the bigger the time needed to proccess. Fairly slow as expected from HDD and 4 scripts running at the same time.
So my question here is:

Does the results of T3-T4 mean, that the file_read_contents and file_put_contents are not reliable for this type of job? To me it looks like they simply do not read the data from file but they are copied from buffer, which means, that old data are saved, not the current data been changed by concurent script. I would welcome more information. I spent a lot of time searching for answers but did not found clear answer. I did this tests because I need proofs. You man want to use my scripts but I am not sure if can I paste here the 6 scripts?

 Now I will add just the fflush test number 7 which is most useful.



PHP 
clearstatcache();
$_DEBUG_ = false;

echo "Lock and flush tester.".time()."
";
die;

while ( time()<1570787996 )
 {
 usleep(500);
 }


function test($n, $p, $_DEBUG_){
  $sname = "$n";    // source
  $tname = "$n.txt";// target
  echo "

$n at "

.time()."
"; for ($i = 0; $i<50; $i++ ){ $start = microtime(true); clearstatcache(); // needed for filesize and touch $st = stat("$sname"); $original_size = $st['size']; if ( $_DEBUG_ ) echo "; 1) prevAccess by ".$st['mtime']." fsize ".$st['size']."; "; $fsize = filesize($sname); if ( $original_size <> $fsize ) die("; fsize total FAILTURE; "); if ($fsize === 0) echo "! The fsize is 0: stat(): ".$st['size']." ;"; else { // READ OPERATION AND LOCK FOR SHARE $locked = false; for ($c = 0; !$locked; $c++): if ( $c > 400) break; $fp = fopen($sname, "r"); $locked = flock($fp, LOCK_SH); if ($locked) break; else { echo "failed to get LOCK_SH;
"
; usleep(5000); } endfor; $s = fread($fp, $fsize ); $success = flock($fp, LOCK_UN); if ( $success === false ) die("; r flock release failed; "); $success = fclose($fp); if ( $success === false ) die("; fclose failed; "); // 10 - data loaded , $p - browser if ( $success ) { $result = touch("$sname",strlen($s),$p); if ( $_DEBUG_ ) echo "; TOUCH: $result;"; } else die("fclose FAIL."); if ( strlen($s)<60 ) echo "*$s LENGTH:".strlen($s)."
"
; } clearstatcache(); $st = stat("$tname"); if ( $_DEBUG_ ) echo "; 2) prevAccess by ".$st['mtime']." fsize is ".$fsize."; "; // WRITE OPERATION WITH LOC_EX $fp = fopen($tname, "w"); $locked = false; $locked = flock($fp, LOCK_EX); if ( $locked ) { // acquire an exclusive lock $success = fwrite($fp, $s); if ( $success === false) echo "; w FAILED;"; else if ( $_DEBUG_ ) echo " $success B written; "; $success = fflush($fp);// flush output before releasing the lock if ( $success === false ) echo "; flush FAILED; "; $success = flock($fp, LOCK_UN); // release the lock if ( $success === false ) echo "; release FAILED; "; $success = fclose($fp); if ( $success === false ) echo "; fclose FAILED; "; clearstatcache(); // needed for filesize and touch $fsize = filesize($tname); if ($original_size>$fsize) { echo "; WRITE FAILED, restoring;"; $original_fname = "$n"; $result = copy($original_fname, $tname); if ($result == false ) die(" TOTAL FAILTURE: copy failed."); else echo " RESTORED;"; } else { if ($fsize === 0) echo "! THE FILE WAS NOT WRITTEN: data length: ".strlen($s)." fsize: $fsize RESOURCE: $fp
"
; if ( $success ) touch("$tname",$fsize,$p); } } else { echo "Couldn't get the lock!"; } $time_elapsed_secs = microtime(true) - $start; if ( $time_elapsed_secs === 0 ) echo " FAILED "; echo "time: $time_elapsed_secs s
"
; } } switch ( $_SERVER['HTTP_USER_AGENT'] ): // FF 1: case "Mozilla/5.0 (Windows NT 5.1; rv:49.0) Gecko/20100101 Firefox/49.0": $p = 1; break; // Chrome: case "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36": $p = 2; break; // OPERA: case "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 OPR/36.0.2130.80": $p = 3; break; endswitch; copy("523","523.txt"); copy("948","948.txt"); copy("1371","1371.txt"); copy("1913","1913.txt"); copy("2701","2701.txt"); copy("4495","4495.txt"); copy("6758","6758.txt"); test("523",$p,$_DEBUG_); test("948",$p,$_DEBUG_); test("1371",$p,$_DEBUG_); test("1913",$p,$_DEBUG_); test("2701",$p,$_DEBUG_); test("4495",$p,$_DEBUG_); test("6758",$p,$_DEBUG_); die; echo "php: " . phpversion(); ?> PHP echo "php: " . phpinfo(); ?>
 
Notice: 523 means the filesize is 523 kB.


You may want to enable $DEBUG option to monitor each proccess. Note: The touch maybe do not work correctly always.
Note: This is not a request for test, this is just request for review.


Also: Please do not be confused by the yellow color curve. There are two yellow colors. The T4 yellow is almost no visible on the diagram because it has very low values.

If you want to solve this question, follow the topic here.

<50 1="" and="" by="" clearstatcache="" echo="" filesize="" for="" fsize="filesize($sname);" i="" if="" mtime="" needed="" original_size="" prevaccess="" size="" sname="" st="" start="microtime(true);" touch=""><60 br="" echo="" length:="" s="" strlen="">

Žádné komentáře:

Okomentovat

GPT: Ramdisk a extrakce .deb

  Následující skript provádí popsané kroky, využívá RAMdisky různých velikostí podle potřeby a provádí extrakci. Tento skript je psán v Bas...

Štítky

.profile adm administrace Adobe AI akcelerace alfa transparence AND any aplikace apt ar archiv asociativní pole atomicity audio autentifikace awk balíčkovací systém bash beacon beacon_hint benchmark Bézierovy křivky bezpečnost biblehub BJT boolean buffer buffering Cache-Conrol Cloudflare code Collector Cut-off ColorManager colorpicker common compare config cookies CPU CPU pipe css CSS3 curl cut čas data loss data lost data transfer reliability datasheet datetime.strptime deb deb-systemd-helper debian debián development dioda diody dpkg dpkg -S dpkg-deb drivers EBO Emitter Cut-off Current eps ETag exclude exec Expires extrakce jediného extrakce názvu balíčku souboru extrakce souboru .deb fflock fflush ffmpeg FIFO file read file write file_get_contents file_get_contents/file_put_contents file_put_contents filter find first_install.sh flock Fly-back dioda font-face fóra fotorezistor fread functions funkce fwrite gate gate drive GDVfs gedit gedit-common geolokace Ghostscript GIO glib gnome gnome settings GNU Privacy Guard gnupg gpg gradient-background grafika grep grep -v groupadd grub grub update gs gsettings gtk gtk.css gtk+ hebrejština history hlavičky HS html html 5 https hudba charakterizace chroot chyba ICES IGBT img sizes img srcset impedance implementace imshow inference inkscape inrush current install jalový výkon javascript javescript jednocení seznamů js kapacita součástek koeficient zesílení komponenty xFce komunikace se serverem konfigurace Krita KSF Last-Modified lazy caching led LEFT JOIN librosa ligatury light-locker lightdm linux list log maják manuál maskování maskování služby masky matplotlib Max-Age measure memory měření MFCC MFCC koeficienty mint Mint 21.3 Mint xFce míry modules moralizace morphologie MOSFET mount moviepy mysql náběhový proud napěťová ochrana návod nel Network Error Logging NLP not Notifications NTFS nth-child oblékání ochrana okruhy přátel OpenVINO IR formát oprava oprava balíčku optočlen org.gnome.desktop.screensaver org.gnome.nm-applet ořezové masky OSHB otázky otázky_jazyky otázky_moralismu_řešení ovladače panely parsování path personifikace photorec php php 4 php 5 php 6 php 7 php 8 phpbb phpBB3 PipeWire pitch PN přechody pnp pole Policykit postscript práva profilování program prune průraz přeinstalování přepěťová ochrana přepolování příkazy připojení k síti připojení k wifi pseudokódd pstoedit PulseAudio PWM regulátory pydub python python3 pytorch ramdisk RBE RDSon read reaktance rectifier regulace vstupního napětí reinstall relyability remount replace restore reverzní geolokace RIGHT JOIN rm role rozvržení disků pro OS linux a data databází řešení samba sdílení sdílení souborů Sec-Fetch-Dest Sec-Fetch-Mode Sec-Fetch-Site Sec-Fetch-User Secure Shell sed Set Cookie show-manual-login show-remote-login shunt schemas schémata schottka skript skupiny sloupce služby song sort soubory soundfile spínané zdroje spínání splines split správa diskových zařízení SQL ssh stabilizace napětí stahování stream string strojové učení stropové učení supplicant svg syntax systemctl systemd-logind T5 tabulky Tangentové úsečky tar témata tepelná ztráta test text-shadow themes thermal runaway time timestamp tkinter tr transistor transition tranzistor tranzistory tuple tvorba otázek TVS ubuntu účiník udiskd udisks unconfined underrun unity-greeter update usermod uživatelé va charakteristika vala vektorová grafika Vgs video Vth vynechání adresářů vývoj while wpa wpa_supplicant wrapovací funkce x xandr xapp-watt xargs -I xed xed-common xfdesktop xml XOR Xorg Xorg Thumbnails xrandr závislosti zdánlivý výkon zdroj zenerka zenerovo napětí zip zip archiv zkratky zpomalení zpracování textu Žalmy