When Celery CachedFileBackend used with filesystem storage (django.core.files.storage.FileSystemStorage), everything works fine.
But there are issues with storages.backends.s3boto3.S3Boto3Storage (and it's fix from #391), as well as with django_s3_storage.storage.S3Storage.
Exception was:
```
Traceback (most recent call last):
...
File "/src/django-imagekit/imagekit/cachefiles/__init__.py", line 131, in __bool__
existence_required.send(sender=self, file=self)
...
File "/libs/utils.py", line 380, in on_existence_required
file.generate()
File "/src/django-imagekit/imagekit/cachefiles/__init__.py", line 94, in generate
self.cachefile_backend.generate(self, force)
File "/src/django-imagekit/imagekit/cachefiles/backends.py", line 136, in generate
self.schedule_generation(file, force=force)
File "/src/django-imagekit/imagekit/cachefiles/backends.py", line 165, in schedule_generation
_celery_task.delay(self, file.generator, force=force)
...
File "/lib/python3.6/site-packages/kombu/serialization.py", line 221, in dumps
payload = encoder(data)
File "/lib/python3.6/site-packages/kombu/serialization.py", line 350, in pickle_dumps
return dumper(obj, protocol=pickle_protocol)
kombu.exceptions.EncodeError: can't pickle _thread._local objects
```
Cachefile strategy may be configured to generate file when file existance required.
To generate images, async backends passes `ImageCacheFile` instance to worker.
Both celery and RQ calls `__repr__` method for each argument to enque call.
And if `__repr__` of object will send `existnace_required` signal, we will get endless recursion.
Issue: #434
In a processes that generates many images, you could run into a
cituation with too man files being open. This results in:
IOError: [Errno 24] Too many open files
This prevents extra IO. Different defaults are used for async backends
since we can’t assume that `existence_required` resulted in existence
synchronously.
StrategyWrapper was unnecessarily implemented as a LazyObject
and it triggered a bug in Django (issue 21840).
Changing the lazy object to a function works just as well and
bypasses the bug.
* 'python3' of https://github.com/vstoykov/django-imagekit:
Add Venelin Stoykov to AUTHORS
Improve logic of contributing ImageSpecFields to Models
Use force_bytes from imagekit.lib in test_cachefiles
Remove @vstoykov's note. Seems like the right place to me (:
Move force_bytes into lib module
Don't use a raw string with \u escapes
Fix sanitizing cache key under Python 3
Add module to sys.modules
Test for Python 3
Insert importer at beginning of list
Delay Django import until needed
Add Python 3 suport and drop support for Python 2.5
Conflicts:
imagekit/cachefiles/__init__.py
As discussed in #214, source_created and source_changed didn't really
have clear definitions. In truth, their names and separation betray
their origins as model receivers in earlier versions. The "source group"
abstraction helped us get away from thinking about things exclusively in
terms of models, but these remained as an artifact.
Differentiating between when the generated file content is required and
when the generated file is just required to exist gives us more
flexibility with strategies.
Reading from an `ImageCacheFile`, will result in accessing its `file`
attribute repeatedly which would result in the `before_access` signal
being dispatched, which in turn would result in many unnecessary calls
to the image cache backend. With this change, we don't send
`before_access` if the file has already been created.
Similarly, we don't need to try to generate the image if we know for
certain that it's already been generated (because we have a reference
to it).
Changed my mind about 04aa72c1f9. It's
just a better description, even if different strategies can change the
behavior so it isn't really very cache-like.
