How to download a delayed file via HTTP in Ruby? -
i use following ruby function download various files via http:
def http_download(uri, filename) bytes_total = nil begin uri.open( read_timeout: 500, content_length_proc: lambda { |content_length| bytes_total = content_length }, progress_proc: lambda { |bytes_transferred| if bytes_total print("\r#{bytes_transferred} of #{bytes_total} bytes") else print("\r#{bytes_transferred} bytes (total size unknown)") end } ) |file| open filename, 'w' |io| file.each_line |line| io.write line end end end rescue => e puts e end end
i want download files (csv, kml, zip, geojson) this website. however, there some kind of delay set up. when click download link in browser takes bit until download window appears. suppose file needs processed on server before can served.
how can modify script take delay account?
i running ruby 2.2.2.
here modification according post , comment:
require 'open-uri' def http_download(uri, filename) bytes_total = nil index = 1 begin open( uri, read_timeout: 500, content_length_proc: lambda { |content_length| bytes_total = content_length }, progress_proc: lambda { |bytes_transferred| if bytes_total print("\r#{bytes_transferred} of #{bytes_total} bytes") else print("\r#{bytes_transferred} bytes (total size unknown)") end } ) |io| # if "application/json" == io.content_type if io.is_a? stringio raise " --> failed, server processing. retry request ##{index}" else # tempfile puts "\n--> succeed, writing #{filename}" file.open(filename, 'w'){|wf| wf.write io.read} end end rescue => e puts e return if e.is_a? openuri::httperror # processing error index += 1 return if index > 10 sleep index , retry end end
Comments
Post a Comment