<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1">
<title></title>
</head>
<body text="#000000" bgcolor="#ffffff">
Hello Pavel and everyone,<br>
<br>
I agree that comipler memory barrier is asm volatile(::: "memory").<br>
However, I was talking about the run-time memory barrier to<br>
prevent aggressive out-of-order and speculative execution in the<br>
processor.<br>
<br>
I understand that "sync" is expensive that it's better to be used at<br>
the application level only when necessary according to the flow<br>
of the applcaition. However, what is important as well<br>
is the effective location where 'sync' should be applied.<br>
Actaully POWER4 and up (e.g. POWER5) processors<br>
have 'lwsync', which one can consider to use at the<br>
O.S. level as a memory barrier that provides the same<br>
ordering function as
the sync instruction, except that a load<br>
caused by an instruction
following the <code>lwsync</code> may be performed<br>
before a store caused by an instruction that precedes the <code>lwsync</code>,<br>
and the ordering does not apply to accesses to I/O memory
(memory-mapped I/O).<br>
<br>
Thus, I proposed lwsync for the above porcessors as <br>
memory barrier at the OS level. Thus, users can decide<br>
the locaiton of I/O memory barrier (e.g. eieio for PPC) according<br>
to their own applcaition. For processors which do not support lwsync<br>
, lwsync is treated as sync.<br>
<br>
Back to the principal, where is the effective location for lwsync or
sync ? <br>
More below.<br>
<br>
Pavel Pisa wrote:<br>
<blockquote type="cite"
cite="mid200709080122.45385.ppisa4lists@pikron.com">
<pre wrap="">On Thursday 06 September 2007 11:43, Feng, Kate wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Joel Sherrill wrote :
</pre>
<blockquote type="cite">
<pre wrap="">The memory barrier patch was PR 866 and was merged in March 2006. It is
in all 4.7 versions. It first appeared in 4.6.6.
</pre>
</blockquote>
<pre wrap="">It looks like it, but RTEMS4.7.x still needs patches.
This is not even fixed in 4.77.99.2.
The memory barrier definitely should be fixed in RTEMS4.7.x
before jumping to RTEMS4.8.
Suggestions follow, except I hope I do not miss anything
since I came up with this a while ago.
1) In cpukit/score/include/rtems/system.h:
#define RTEMS_COMPILER_MEMORY_BARRIER() asm volatile(::: "memory")
seems to be wrong and misplaced.
The memory barrier is processor dependent.
For example, the memory barrier for PowerPC is "sync".
Thus, for PPC, it would seem more functional to place
#define RTEMS_COMPILER_MEMORY_BARRIER() asm volatile("sync"::: "memory")
in cpukit/score/cpu/powerpc/system.h
or somewhere in the processor branch.
</pre>
</blockquote>
<pre wrap=""><!---->
Hello Kate and others,
I would like to react there, because I think, that proposed
addition of "sync" is move into really bad direction.
RTEMS_COMPILER_MEMORY_BARRIER is and should remain what it
is, I believe. It is barrier against compiler optimizer
caused reordering of instruction over the barrier.
This does not try to declare/cause any globally visible
ordering guarantee, by name and anything else.
Each architecture 'X' conforming CPU has to guarantee,
that even after complex CPU level instruction reordering,
register renaming and transfers delaying an sequence
of instruction would result in same state (all viewed
from CPU POV) as if instructions has been processed
in sequential order one by one.
This does not mean anything about external memory transfers
order at all (at least for PPC, there are some special rules
for x86 caches for compatibility with old programs).
The macro ensures only ordering of memory transfers
from actual CPU POV/perspective. But this is enough
even for POV of normal mode and consecutively invoked
exception handler working with same data.
Even if exception handler starts and CPU does not finish
transfers caused by previously initiated operations, reads
from exception on same!!! CPU would read back data from
write buffer if the address corresponds to previously written
data. So preemption or CPU IRQ flags manipulation in scope
of the actual CPU does not need enforcing ordering of real
memory by very expensive "sync" instruction. It only needs to
be sure, that CPU accounts/is aware of the value write transfer
before at correct point in the instruction sequence.
On the other hand, there could be other reasons and situations
requesting correct ordering of externally visible transfers.
For example, if IRQ controller is mapped as peripheral into
external memory/IO space and CPU IRQ is disabled, than some
mask is changed in the controller to disable one of external
sources and it is expected, that after IRQ enabling on CPU
level there cannot arrive event from that source, ordering
of reads and writes to the controller has to be synchronized
with CPU ("eieio" has to be used in the PPC case). But it
is not task for CPU level IRQ state manipulation. The ordering
should and in RTEMS case is ensured by IO access routines
which include "eieio" instruction. On the other hand, if
some external device is accessed through overlay structures
(even volatile), then ordering could be broken without
explicitly inserted "eieio".
Other legitimate requirement for strict ordering/barrier for
external accesses are the cases, where external device/DMA/coprocessor
accesses/shares data in system/main memory with CPU.</pre>
</blockquote>
The share data among multi-threads needs memory barrier<br>
as well. Thus, the semaphore used for synchronization between two
different<br>
thread needs it as well. At cpukit/rtems/src/semrelease.c, the 4.7..x
OS did not wish<br>
the compiler to be out-of-order at that point before
_Thread_Enable_dispatch().<br>
However, does it make sense to allow the run-time system memory access<br>
out-of-order until the code reach the 'sync' or 'lwsync' at the user
level ? Perhaps,<br>
those who understand all levels of OS will know better about the
answer to this. <br>
Logically, I am a little bit confused.<br>
<blockquote type="cite"
cite="mid200709080122.45385.ppisa4lists@pikron.com">
<pre wrap="">
The "sync"/cache range invalidation/flushing is required before
and after external memory accesses (the exact details depend on
transfers directions and other parameters).
</pre>
<blockquote type="cite">
<pre wrap="">3) Among PPC shared/irq/irq.c and other PPCs,
_ISR_Disable( _level ), and _ISR_Disable( _level )
should be used instead of _CPU_ISR_Disable(level) and
_CPU_ISR_Enable( _level )
</pre>
</blockquote>
<pre wrap=""><!---->
But I fully agree with you, that sequences like following
one are fatally broken
_CPU_ISR_Disable(level);
*irq = rtems_hdl_tbl[irq->name];
_CPU_ISR_Enable(level);
There is no guarantee, that operation which has been expected
to be protected would not be moved outside of protection sequence.
Explicit or implicit RTEMS_COMPILER_MEMORY_BARRIER is missing there.
</pre>
<blockquote type="cite">
<pre wrap="">Actually, I think a better one should be rtems_interrupt_disable(level)
and rtems_interrupt_enable(level).
</pre>
</blockquote>
<pre wrap=""><!---->
The code should be changed according to one of your proposals.
</pre>
<blockquote type="cite">
<pre wrap="">2) In order for the inline to work, the
CPU_INLINE_ENABLE_DISPATCH should be defined to be TRUE.
Thus,
in cpukit/score/cpu/powerpc/rtems/score/cpu.h:
-#define CPU_INLINE_ENABLE_DISPATCH FALSE
+#define CPU_INLINE_ENABLE_DISPATCH TRUE
</pre>
</blockquote>
<pre wrap=""><!---->
As for the non-inlined version of _Thread_Enable_dispatch,
there should not be problem. Calling function without static
or inline attributes is considered as full compiler memory ordering
barrier point. So no explicit compiler barrier should be needed
there.</pre>
</blockquote>
I agree with Sergei's vote.<br>
<br>
PS. Does anyone know how to find the number of OPcodes for all the<br>
PPC assembly code ?<br>
<br>
Reagrds,<br>
Kate<br>
<blockquote type="cite"
cite="mid200709080122.45385.ppisa4lists@pikron.com">
<pre wrap="">
All this is based upon my understanding of code and computer
systems principles. There is no doubt, that there could be
many other problems and errors. But if there are problems
with IRQs behavior on PPC, then the checking, that sequences
like above one do not exist. The _ISR_Disable()/_ISR_Disable()
or higher level rtems_ variants should be used in noticed source
file. Else bad things could happen.
Excuse me for long answer, but I wanted to clarify things
as good way as I could.
Best wishes
Pavel
</pre>
</blockquote>
<br>
</body>
</html>