How to download multiple files in the background with tsNet
Introduction
In order to provide a smooth user experience, it is often helpful if any files (images, videos or other data files) that need to be retrieved across the network can be downloaded by the application seamlessly in the background.
The following lesson shows how you can download a series of files in parallel and keep of track of which ones are yet to be retrieved.
Lay out the Stack
Create a new stack then drag a button and a field onto it.
Set the name of the button to "Download Files" and the name of the field to "Status".
Save this stack.
Create the Script
Edit the script of the button that you placed on the stack by right clicking on the button and selecting "Edit script".
Add the following code.
local sMaxRequests
local sActiveRequests
local sDownloadArray
local sRequestedArray
local sDataDir
local sDownloadErrors
-- The JSON data file must be in the following format:
--
-- [
-- {
-- "URL": "<url to download>",
-- "status": "",
-- "filename": "<filename to save download to>"
-- },
-- {
-- "URL": "<url to download>",
-- "status": "",
-- "filename": "<filename to save download to>"
-- }
-- ]
--
on mouseUp
-- Make sure we are not already downloading files in the background
if sActiveRequests is not 0 then
answer "Downloads currently in progress."
end if
-- Set a limit for the number of requests we want to perform at a time
put 3 into sMaxRequests
-- This will be set to true if an error occurs while downloading any of the files
put false into sDownloadErrors
-- Reset the array of requested downloads
put empty into sRequestedArray
-- Specify the folder that downloaded files will be stored in
put specialFolderPath("Desktop") into sDataDir
-- Retrieve the JSON data file which contains the files to download and the current download status of each
-- and convert it to an array called sDownloadArray
put URL("file:" & sDataDir & "/URLs.json") into tJsonData
put JSONToArray(tJsonData) into sDownloadArray
put "Downloading..." into field "Status"
-- Begin downloading
send "downloadURLs" to me in 0 milliseconds
end mouseUp
on downloadURLs
put the number of lines in the keys of sDownloadArray into tArrayCount
-- If we are not already downloading the maximum number of simultaneous files, loop through
-- sDownloadArray and initiate a request to an entry that is not yet "downloaded"
if sActiveRequests < sMaxRequests then
repeat with x = 1 to tArrayCount
if sDownloadArray[x]["status"] is not "downloaded" and sRequestedArray[x] is not true then
put tsNetGet(x, sDownloadArray[x]["URL"], tHeaders, "transferComplete") into tResult
-- Increment the number of active requests and flag this item as having been requested
if tResult is empty then
add 1 to sActiveRequests
put true into sRequestedArray[x]
end if
end if
-- Stop downloading new files if we have reach the maximum number of simultaneous downloads
if sActiveRequests = sMaxRequests then
exit repeat
end if
end repeat
end if
-- If we have looped through the array and no downloads are occurring, we have finished processing
-- all the files to download, so execute the "downloadsCompleted" handler
if sActiveRequests = 0 then
downloadsCompleted
exit downloadURLs
end if
-- Keep looping if we are not yet finished
send "downloadURLs" to me in 50 milliseconds
end downloadURLs
on transferComplete pID, pResult, pBytes, pCurlCode
-- Decrement the number of active requests as one has just finished
subtract 1 from sActiveRequests
-- Check the status codes and look for success
if pCurlCode is 0 and pResult is 200 then
-- If successful, retrieve the downloaded data
put tsNetRetrData(pID, tError) into tData
if tError is empty then
-- Downloaded data has been retrieved successfully, save the data to the filesystem
-- and update the status of the URL in sDownloadArray
put "downloaded" into sDownloadArray[pID]["status"]
put tData into URL("file:" & sDataDir & slash & sDownloadArray[pID]["filename"])
-- Save the array back to the filesystem so that if the application is closed prior
-- to all files being downloaded, we can pick up where we left off
put ArrayToJSON(sDownloadArray,,true) into URL("file:" & sDataDir & "/URLs.json")
end if
else
-- A transfer must have failed, flag the failure and update the URL's status in sDownloadArray
put true into sDownloadErrors
put "failed" into sDownloadArray[pID]["status"]
put ArrayToJSON(sDownloadArray,,true) into URL("file:" & sDataDir & "/URLs.json")
end if
-- Always call tsNetCloseConn to release any memory associated with the transfer
tsNetCloseConn pID
end transferComplete
on downloadsCompleted
-- Check for any download errors during processing, and notify the user accordingly
if sDownloadErrors is true then
put "All downloads attempted, but at least one was unsuccessful. Click the button to retry any failed downloads." into field "Status"
else
put "All downloads completed successfully." into field "Status"
end if
end downloadsCompleted
Create a file
The button script above accesses a text file to retrieve and maintain the list of URLs that must be downloaded.
Open up your favourite text editor program and copy the following text into your file. Name this file "URLs.json" and save it to your desktop.
[
{
"URL": "https://www.livecode.com",
"filename": "livecode.txt",
"status": ""
},
{
"URL": "https://www.google.com",
"filename": "google.txt",
"status": ""
},
{
"URL": "https://www.microsoft.com",
"filename": "microsoft.txt",
"status": ""
},
{
"URL": "https://www.apple.com",
"filename": "apple.txt",
"status": ""
}
]Test
Now you are ready to test your coding and see the results.
Switch to Run mode and click the button.
More information
This example uses a text file in JSON format to maintain the list of URLs to download and their current download status. However, this information could be stored in any number of formats. It could also be retrieved from a remote URL or stored in a database.
Mike Felker
I am having a problem.
I need to achieve this without server side scripting. I need to download multiple files all within the same livecode program. How would I do this?
Right now, if I try to download files from a server (good internet connection and working server assumed) using SFTP with a repeat / end repeat loop, it works sporadically. Unless I issue a answer box in the loop, it may or may not reliably download each small file (under 5k each). If I issue a wait timer of some kind, it does not work. However, if I issue a answer box (like answer "The file has been downloaded" with "OK" - that makes it work). Is there some command I am missing that allows files to be downloaded in the background without user interaction to be successfully downloaded every single time?
Thanks.
Mike
Matthias Rebbe
You are using tsNET, right. I´ve ran into a similar problem and solved it by setting
tSettings["no_reuse"] to TRUE, which forces tsNet to disconnect the connection to the server after each transfer.
The dictionary says
"no_reuse": Set to true to specify that the connection to the server should be disconnected and not left open for any future connections.
At least this helped me to solve it, but you could give it a try.
Mike Felker
I tried this and it did not work. I added:
put true into tSettings["use_ssl"]
put true into tSettings["no_reuse"]
put true into tSettings["save_sent_headers"]
but it still cycles through the loop too fast and does not download anything.
Mike
Mike Felker
Wait timers don't work either. I tried:
wait 5 seconds
in the loop and it does wait 5 seconds, but does not initiate a download.
What is weird is that if I replace that line with:
answer "Whatever text I want" with "OK"
then it works fine.
Mike
Matthias Rebbe
Hm, so let´s hope that Charles Warwick as the developer of tsNET or Eleanor chime in. Maybe they have an idea. The people at Livecode Ltd. are back to work next week. Could you post the script, of course without any real login or url data, one could test? I would like to try with my server.
Mike Felker
I am getting a "ID already in use error". How do we clear the ID before each new download in the loop?
Mike
Mike Felker
I can only post a bit of it - we have patents pending.
Server variables and username/password variables are ommitted.
Assume there are 10 files:
[code]
-- Download from server
-- Store the SFTP server details in some variables
put "sftp://myserver" into tSFTPServer
put "/path/to/file" into tLocalRegFile
put "/path/to/user/file" into tUserFile
put true into tSettings["use_ssl"]
put true into tSettings["no_reuse"]
put true into tSettings["save_sent_headers"]
repeat with x = 10 down to 1
## If we make the below line active, then it works as long as you wait long enough for the file to download
--answer "Downloading Block " & x
## Create files to download
put tUserFile & "_" & x & ".bl" into tRemoteDownload
put tLocalRegFile & "_" & x & ".bl" into tLocalDownload
put tsNetGetFile("mySFTPdownload", tLocalDownload, tSFTPServer & tRemoteDownload, tHeaders, "tsNetGetFileFinished", tSettings) into tResult
if tResult is not empty then
-- Inform the user of the error
put "There has been an error. Error details:" && tResult into field "fldStatus"
## We could warn with a answer box
--answer "There has been an error"
else
## Download was a success
put "Success" into field "fldStatus"
end if
end repeat
end opencard
## TSNet required code
on tsNetGetFileFinished pID, pResult, pBytes, pCurlCode
put "tsNetGetFile request has completed, checking result - code: " &pCurlCode
-- If pCurlCode is anything other than 0, the request has not been successful.
-- pResult will also be 0 for successful SFTP transfers
-- If either of these are not the case, we will inform the user
if pCurlCode is not 0 or pResult is not 0 then
answer "Could not retrieve data from site"
else
-- The file has been downloaded and stored in the location indicated by the tLocalPath variable
put "File has downloaded successfully" into field "fldStatus"
end if
-- Tidy up the resources used by the tsNetGetFile call:
tsNetCloseConn pID
end tsNetGetFileFinished
[/code]
That's what I can show. Thanks for any help.
Mike
Matthias Rebbe
Which tsNet command are you using? The connectionID, which is needed for non-sync tsNet commands, is user defined. So you have to define an ID for each download request. How does your script looks like?
Mike Felker
put tsNetGetFile("mySFTPdownload", tLocalBlockToDownload, tSFTPServer & tRemoteBlockToDownload, tHeaders, "tsNetGetFileFinished", tSettings) into tResult
Mike Felker
How would I assign a ID in the loop?
Mike
Mike Felker
put tsNetGetFile("mySFTPdownload", tLocalBlockToDownload, tSFTPServer & tRemoteBlockToDownload, tHeaders, "tsNetGetFileFinished", tSettings) into tResult
How would I set up a new ID within the loop?
Mike
Matthias Rebbe
Currently your connectionID is always mySFTPdownload
You could change
put tsNetGetFile("mySFTPdownload", tLocalBlockToDownload, tSFTPServer & tRemoteBlockToDownload, tHeaders, "tsNetGetFileFinished", tSettings) into tResult
to
put tsNetGetFile(x, tLocalBlockToDownload, tSFTPServer & tRemoteBlockToDownload, tHeaders, "tsNetGetFileFinished", tSettings) into tResult
so that the connectionID now contains the value of X, which is the loop counter or however this might be called.
Mike Felker
You are AWESOME Matthias! All fixed. Works perfectly with a 2 second wait command.
Thank you!
Mike
Matthias Rebbe
Great to hear that it works now.