[IPOL discuss] original files, filenames in templates and make -j [Fwd: Re: Questions publication Ipol LSD]

[Nicolas Limare]

On Wed, Dec 8, 2010 at 12:12:41 -0200, rafael grompone von gioi wrote:
> >
> > Le fichier original est supprimé par la démo. En as-tu vraiment
> > besoin?
> 
> Je trouve intéressante d'avoir les fichier originales uploades.
> Par fois le gens envoyent des images intéressantes pour
> des autres raisons et je trouve bien avoir acces à l'image originale.
> Aussi, ça peut permettre de penser à changer le pre-processing
> si on voit quelque chose que ca ne va pas.
> 
> D'ailleurs, dans la version actuel du demo LSD les fichiers originales
> sont gardes, et je trouve ca tres bien.
> 
> Je pense que on devrai pouvoir les gardes.

I decided to delete the original images in the new demo system because
some users upload HUGE files (5-10Mb), whose size gets even bigger on
the disk once we convert the original file format (unknown, usually
JPEG) to lossless PNG. I didn't see the point of keeping them and
using disk space[1] while only a downsized version was processed.

I will try to change it and keep the original files as
input_0_orig.png, input_1_orig.png, ..., but the long-term
availability of these files will depend on the disk space issues.

[1] especially multiple backup disk space, and CPU time and network
bandwidth for backup sync

> > Tu passes ces infos en paramètre du template
> >
> > self.tmpl_out(....
> >              eps="foo.eps",
> >              svg="bar.svg")
> >
> > Tu les récupère dans le template
> >
> >  ${app.work_url + eps}
> >  ${app.work_url + svg}
> 
> Je vais essayer aussi.

Note: this still works, but I just changed the way file names are
handled in templates, because passing file names as parameters didn't
make much sense, these file names are the same every time the template
is called, and each template is only used by one demo, so it was just
lots of useless and redundant code lines.

Instead of...

app.py:
  self.tmpl_out("result.html",
                eps="foo.eps",
                svg="bar.svg")
result.html:
 <a href="${app.work_url + eps}">eps file</a>
 <a href="${app.work_url + svg}">svg file</a>

... you can just use ...

app.py:
  self.tmpl_out("result.html")
result.html:
 <a href="${app.work_url + "foo.eps"}">eps file</a>
 <a href="${app.work_url + "bar.svg"}">svg file</a>

> >> 6) Tu as mis dans la section build de axpb un appel à "make"
> >> avec une option "-j4". Est-ce que c'est nécessaire? Pour quoi?
> >
> > -j4 permet de compiler plus vite (4 process en parallèle) les codes
> > composés de plusieurs fichiers. Compiler plus vite reduit la durée
> > pendant laquelle la démo est indisponible. Si le machine n'a qu'un
> > CPU, cela n'accélère pas mais ne ralentit pas beaucoup non plus.
> 
> Ok, je le laisse, moi aussi.
> Dans ce cas, pour quoi ne pas utiliser simplement "-j",
> comme ca make utilise autents de CPU que disponibles?
> (si j'ai bien compris : )

Not exactly. "-j" means "use as many processes as you want", not "use
as many process as it makes sense to".

If a makefile involves compiling 500 .c source code files
into 500 .o object files, then linking these 500 .o object files into
one executable, then "make -j" will launch 500 process for the .c->.o
compilations, then one process to link once all the compilation
processes are over. This will not be efficient, the system will handle
it but will be overloaded by 500 process competing to access the
2,4,... or 64 available cores and the disk.

A solution would be an environment variable with the number of
available cores, but a rebust code (working even when this
variabel is not available) would imply some preliminary tests and
would obscure the code. So, I think "-j 4" is a reasonable compromise,
will be OK even on sungle-core machines, and will fasten compilation
of typical programs (there is no code with 500 files in IPOL).

-- 
Nicolas LIMARE
http://nicolas.limare.net/                         pgp:0xFA423F4F
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://tools.ipol.im/mailman/archive/discuss/attachments/20101209/25aa3a71/attachment-0001.pgp>