4

1. The problem:

Series upload of 100 or more files via FTP is interrupted. Behavior irregular. On a new PC from 2011, Ubuntu just installed. No other problem with Ubuntu. Fast browser action and Website loading.

QUESTION What could I try, in addition to the steps described here below?
Or is there a primitive error or a primitive solution?

Original text not modified. Updates done by adding:

  • Dec.7, 2011: Information added below in 3.2)
  • Dec.9,2011: added: 4.d) 4.e)
  • Dec.10,2011: added 4.f)

2. More facts:

FTP upload of several 1000 files (several sites) is here done by own Perl programs which send a series of FTP standard commands to the standard ftp of the LINUX system. Effect: Similar to mirroring - but enables fine-tuning.
The used single commands "ftp mput" comprise e.g. 100 files, each file approx. 100 KB. Approx. 1000 files per Website.

Uploading to: Websites using shared hosting. The servers use Linux, e.g. Bluehost=Hostmonster or Lunarpages.

3.1) So far I never had such a problem.

During years I never had such a problem with Fedora. Fedora 13 on a PC from 2006 continues to do the job even now properly (idential DSL connection, identical DSL hardware). I intentionally did not upgrade Fedora 13. So LINUX ftp might have changed a bit in the meantime. But it is not probable that this is the reason.

PCs are here only used as OS supplier. Data and own software are on an external HD, are OS-independent and portable (organized by Perl software).

The new UBUNTU PC does not do this job perfectly, while working on the identical external HD and with an identical software environment.

(Only the network cable differs, but this should not cause any problem. 1 m length for the old FEDORA PC, 5 m for the new Ubuntu PC.)

3.2) UBUNTU specifics cause probably the problem (added Dec.7,2011)

In the meantime / now I have :

UBUNTU 64b from late 2011 on the new PC / 2011, SATA hard disks.

UBUNTU 32b from late 2011 on the old PC / 2006, PATA (/IDE) hard disks.

These are just suppliers of the operating system. All data, software and execution is done on an external HD which is IDENTICAL for both cases.

In both cases, the described problem occurs. It does NOT occur for the old PC when run with Fedora 13 (from 2009 or so).

The problem is hence very probably somehow correlated to specific features of UBUNTU.

There is a very low probability that since FEDORA 13 the general LINUX system has changed, resulting in this problem.

With a rapid Google search, so far I did not find any information about similar problems.

Everything else of UBUNTU is working fine on the new PC with a wide screen. So it is from now for a while here the choice.
I am getting these weeks a 10 times faster DSL Internet connection. Perhaps the problem will then disappear. (I suppose it will NOT disappear.)

4.a) Priority setting does not help.

With the command nice I have tested a priority of -18 (verified in gnome-system-monotor - yes, -18). This did not help.

4.b) sudo use did not help

I also tried to call the Perl program which does the job, with sudo. This did not influence the result.

4.c) Behaviour irregular

There is no rule at what point of time (which file) the upload is stopped. Normally after some minutes - only once it did the whole job of 2 hours. There is perhaps a minor correlation to the rush hours of Internet use. But this is not sure.

4.d) No help from: source code, -i flag, -v flag (added 9 Dec.2011)

The involved COMPLETE long Perl programs would not be helpful here. (Various features, sites, specifics,...).

Here the OWN command (Perl subroutine) with the upload problem (example):

&FTP_c_mput (" www/ppp-de/*.htm")

This simply does the FTP command : mput www/ppp-de/*.htm

for ~200 files, but (the problem:) halts at file 30 or so

-i All is automatic - hence already with -i flag, hence nothing interactive. So a timeout error should normally never occur.

-v --- Always in verbose mode (therefore the results in nr.5.2. and 5.3.)

I still have to implement within the software the debug function -d (like recommended). It is doubful that it will help - because...:

4.e) Most probable problem reason - I suppose (added Dec 9,2011)

Most probable is that some process, specific to Ubuntu, creates a delay - so that the ftp program does not supply fast enough the next file to the remote PC (Internet server). But I did not state delays of several seconds for any other function. Everything works perfect and fast. And I could not find any correlation of some software action or so before the problem occurrence.

I state per,anently a short hard disk access every 3 seconds (if the control led is acting properly). Every 3 seconds - or faster. Not regular. This may have simple explanations. This even continues when there is no application running - a totally blank monitor. But I do not believe that the reason is related to this.

4.f) Debug feature? ftp -d (added Dec.10,2011)

The -d flag for ftp : Not tried (while recommended by a comment).

The -d flag as such is not sufficient. To get ftp logged, various steps are required (rsyslog.conf...). The probability that this time investment would lead to success, is for this problem type close to zero.

So I will instead continue the work-around: Doing bulk uploading on Fedora (= PC 1), all the rest on Ubuntu (= PC2), and try other steps to find in some future the solution.

5.1. Example: This is a control display during uploads

This display is done by my own Perl programs, hence not by FTP. Shortened. May include 100...300 files.

www/xmed-ppp-de/index.htm www/xmed-ppp-de/wweb-pare-med-de.htm www/xmed-ppp-de/wwee-fina-med-de.htm www/xmed-ppp-de/wwfu-cont-med-de.htm www/xmed-ppp-de/wwfu-sepa-med-de.htm

5.2. Example: The standard display of the ftp program is like:

150 Connecting to port 63555

#...

226-File successfully transferred

226 7.126 seconds (measured here), 10.14 Kbytes per second

73985 bytes sent in 0.81 secs (89.7 kB/s)

local: www/xmed-ppp-de/wyck-tob-bo-med-de.htm remote: www/xmed-ppp-de/wyck-tob-bo-med-de.htm

200 PORT command successful

5.3. Example: The display when interrupted:

With new installed UBUNTU on a new PC, the series is ending mostly after approx. 30 files, varying in random manner between 20 und 50, with the following error message: - hence: a Timeout error for MY typing - but it was the FTP bulk command mput , - so there can not really be a typing speed problem on MY PC...

150 Connecting to port 63562

#...

226-File successfully transferred

226 10.317 seconds (measured here), 14.02 Kbytes per second

148068 bytes sent in 6.65 secs (21.8 kB/s)

local: www/xmed-ppp-de/wyck-tob-ris-med-de.htm remote: www/xmed-ppp-de/wyck-tob-ris-med-de.htm

200 PORT command successful

421 Timeout - try typing a little faster next time

local: www/xmed-ppp-de/wyck-tob-sto-med-de.htm remote: www/xmed-ppp-de/wyck-tob-sto-med-de.htm

No control connection for command: Success

local: www/xmed-ppp-de/wycu-nut-med-de.htm remote: www/xmed-ppp-de/wycu-nut-med-de.htm

............... etc. etc. for all following files ............

Piotre
  • 61
  • You may wish to call ftp with the -i flag to prevent prompting which may be what cause timeouts. – Anonymous Dec 07 '11 at 23:42
  • You may wish to call ftp with the -d (debug) and -v (verbose) flags to enable debugging so you easier figure out what is wrong. – Anonymous Dec 07 '11 at 23:42
  • Perhaps it would be easier for people to help you if they had access to the source of your Perl script(s). – Anonymous Dec 07 '11 at 23:43
  • for clarity, I added my comments in the man text. All these extensions of the main text are listed on top of the text. – Piotre Dec 10 '11 at 11:43

1 Answers1

2

(Here is my own answer for my own question above.)

The solution is:

The problem does not any more occur when using a faster Internet connection.

It occured for a speed of 0.2 Mb/sec (past DSL standard speed for uploads - level like in 2005,2006; not measured).

It does not any more occur for a speed of 1.2 Mb/sec or similar (current upload speed, via mobile Internet access UMTS, measured with http://www.umts-speedtest.com )

In case of reduced speed with this mobile access form (UMTS), the problem occurs again. This may happen during the daily rush hours of the mobile access network to the Internet.

The speed here will during the next weeks increase to 5 Mb/sec (upload, 25 or 50 for downloads). So this will probably settle the problem in a definite manner.

Now we have a good definition for the problem reason:

This problem type only occurs for the used UBUNTU distribution DVD / late 2011 (but probably for all current UBUNTU distributions).

It does never occur for Fedora Core, at least up to version 13=2009. Later versions have not yet been checked. It is supposed that later and current (2011,2012) versions of Fedora Core will also not have this problem.

The problem is not very important. It is unusual that such bulk upload jobs are done for users with slow upload speeds like typical for 2005 or so.

Bug removal : Is there a bug?

FTP utility programs probably do bulk uploads in a different manner and will therefore probably not be affected. The problem will probably only occur if bulk uploads are done by using for 30++ files the basic FTP command: mput

This command should work for all upload speeds - also for modems with a telephone landline - and for all file quantities.

So is there a bug? - Probably just in the sense of some default initial configuration. The process priority setting did not settle the problem. But it might be that the problem is related to computing priorities for the high level graphics features of current Ubuntu (I learned to like them...).

Perhaps this FTP problem does not matter. Before placing this question on this site, I had looked up via Google I did not find somebody else communicating the same problem type.

Other possible reason?

Some other totally different reason might be: Instability of Internet connections, due to current local changes here,

and with some higher vulnerability from this for the configuration of UBUNTU (from late 2011) than that for Fedora Core (from 2009).

So let us forget about it...

Piotre
  • 61