Sure, as long as you are working with a standard and linear memory model. The assumption that size_t and ptrdiff_t can store a pointer may break down on segmented memory architectures for example. If you want to code against the standard instead of an assumption, use uintptr_t and intptr_t instead.
When would someone want to use intptr_t instead of uintptr_t? Does a signed memory address even make sense? Perhaps intptr_t is available to avoid compiler warnings about mixing signed/unsigned ints when adding a uintptr_t and a (signed) ptrdiff_t.
While I don't know of any specific use-case, it is of course very easy to imagine an architecture with signed memory-address space. One could for example separate protected/kernel memory from user land memory with the signedness.
ptrdiff_t is not guaranteed to be the same size as size_t, but ssize_t is. Also, IIRC ptrdiff_t is unsigned, whereas ssize_t is signed, which could make 32/64 bit arithmetic a PITA.
you are correct inasfar as ssize_t is a posix extension.