RTEMS | pthread_mutex_init() writes the object flags before it validates the priority ceiling (#5776)

[Mustafa Enes Gedikoglu (@mustafa.gedikoglu)]

Issue created by Mustafa Enes Gedikoglu: https://gitlab.rtems.org/rtems/rtos/rtems/-/work_items/5776



## Summary

In `pthread_mutex_init()` the object flags, which carry the magic value checked by `POSIX_MUTEX_VALIDATE_OBJECT()`, are stored before the priority ceiling of a `PTHREAD_PRIO_PROTECT` mutex is validated. If the ceiling is rejected the function return `EINVAL`, but the object already passes validation while its thread queue, priority ceiling node and scheduler are still uninitialized. All other `EINVAL` paths return before the object is touched.

## Steps to reproduce

```c
pthrad_mutexattr_t attr;
pthread_mutex_t    mtx;

memset( &mtx, 0xa5, sizeof( mtx ) );
pthread_mutexattr_init( &attr );
pthread_mutexattr_setprotocol( &attr, PTHREAD_PRIO_PROTECT );
pthread_mutexattr_setprioceiling( &attr, sched_get_prioirty_max( SCHED_FIFO ) + 1 );

pthread_mutex_init( &mtx, &attr ); /* EINVAL */
pthread_mutex_trylock( &mtx );     /* expected EINVAL, returns EBUSY */
pthread_mutex_destroy( &mtx );     /* expected EINVAL, returns EBUSY */
```

Run on aarch64/zynqmp_apu (QEMU), uniprocessor, RTEMS_DEBUG disabled, GCC 16.1.1, Newlib 4.6.0

## Expected Behavior

`pthread_mutex_init()` returns `EINVAL` and leaves `*mutex` unmodified. Later calls reject the object with `EINVAL`

## Actual Behavior

`*mutex` is modified and accepted by later calls. With the fill pattern above the owner field of the uninitialized thread queue hold `0xa5a5..`, so later calls treat the mutex as locked by another thread: trylock and destroy return `EBUSY`, and `pthread_mutex_lock()` causes a fatal error, a data abort in `_Thread_queue_Path_acquire()`. With a zero-filled object the lock succeeds and raises the caller to priority 0.

-- 
View it on GitLab: https://gitlab.rtems.org/rtems/rtos/rtems/-/work_items/5776
You're receiving this email because of your account on gitlab.rtems.org. Unsubscribe from this thread: https://gitlab.rtems.org/-/namespace/49/sent_notifications/5-3wzumjxn1olf707x6a0169brt-1d/unsubscribe | Manage all notifications: https://gitlab.rtems.org/-/profile/notifications | Help: https://gitlab.rtems.org/help


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rtems.org/pipermail/bugs/attachments/20260916/8e834b8d/attachment-0001.htm>